With the absence of a notes.ini entry of MakeDominoGoFast=1, I'll show you how to tune Domino to get valuable performance increases with some simple solutions.
Making Domino run lean and mean is similar to tuning other applications on IBM i. There are things you can do based on your environment to make Domino perform better on some systems that won't make much difference on others. There are also plenty of additional factors at the hardware and operating system level, such as the existing workload of the partition that Domino is hosted on, the amount of disk arms in use, and the communications infrastructure both on the system and attached to the system.
If you have serious processor, disk, or memory contention problems, the solution to speeding up your Domino server may have nothing to do with Domino. If you're running applications that run on top of Domino (like Quickr, Sametime, IBM Notes Traveler), then you must take into consideration the system requirements of those applications. With that being said, you can ensure Domino is set to perform optimally if running on a properly sized system.
The key when adjusting any application performance is understanding what your bottlenecks are, if any, at the system level. A healthy system is your base starting point for healthy applications. IBM Navigator for i offers many ways to see system performance at a high level and drill down to determine problem areas. IBM's Dawn May has some great info on IBM i performance management and the new tools in Navigator. If you're at the COMMON Fall Conference and Expo, be sure to check out her presentations on the topic. A must-see.
What's great about the recent versions of IBM Domino is that they're baked with the ingredients to ensure that it runs as lean and mean as possible. You just need to turn them on.
Document Attachment Object Storage (DAOS)
DAOS is an IBM Domino feature that allows attachments to be stored outside the NSF file in Notes Large Object (NLO) files. DAOS allows multiple copies of the same attachment to be stored as a single referenceable object. By storing it as an NLO, the relatively static attachment data is kept separate from Domino database (NSF) data, improving NSF performance. Let's say you send an email with a 12 MB attachment to 65 people in a distribution group. Without DAOS enabled, the attachment data would equal 12 MB in your NSF's Sent folder and an additional 780 MB of storage consumed across those 65 mail recipients' NSF Inboxes. With DAOS enabled, you add 12 MB to an NLO file. The "attachment" in your Sent items and in the Inbox of your recipients is actually just a symbolic attachment. It still looks exactly the same to the end user, but when they open the attachment it's pulled from the NLO rather than the NSF.
With DAOS enabled, I've seen many customers (myself included) get 30-60% savings on disk space. I've yet to see anyone get less than 30% of Domino-related disk returned. DAOS is real return on investment for taking a little bit of time to move to at least Domino 8.5, and 9.0.1 for additional improvements and fixes. And DAOS doesn't mean you need to bring your mail clients up to 8.5 or above. This is a server-side setup, so you can activate it very quickly. Reducing the amount of data on disk will certainly help performance, especially if you're already seeing disk contention.
When implementing DAOS, you need to set an attachment minimum size in the Server Document. This means that if an attachment doesn't meet or exceed the size requirement, it doesn't get put into a NLO. There's no need to waste resources by putting attachments of 5 KB into NLO. Like any type of performance tuning, it's a little bit of a balancing act. Personally, my minimum size is 1 MB (1048576 bytes in the minimum size field). This is the preferred value for IBM i in the DAOS best practice document; however, you should really run the DAOS Estimator tool to ensure you set the most effective value based on your environment. The DAOS Estimator will run through all NSF files on your Domino server and then give you estimates on how much disk space it can save based on your minimum attachment size.
To implement DAOS:
1. Upgrade the Domino server to a minimum of 8.5 (or just go to Domino 9!).
2. Transaction logging needs to be enabled via the Transaction Logging tab of your server document. Just about every Domino conversation or presentation I see claims that transaction logs need to be on a separate set of drives to ensure that you get proper performance. Some horrendous IBM technotes out there that I've tried to have rectified state the "AS/400s with one logical drive mimicking many drives" won't work—obviously written by someone having very little IBM i knowledge. When you set up transaction logging, have no fear. You can put those transaction logs in a directory right under the Domino server's data directory.
3. Set the "Shared Mail" field to "None" on the Shared Mail tab of your server document and run the object UNLINK command to unlink any existing mail files from the shared ones.
4. Fill in the four fields on the DAOS tab of your Server document:
- Store file attachments in DAOS: Set to Enabled.
- Minimum size of object before Domino will store in DAOS: The default is 4096, but 1048576 is the recommend value for IBM i, as I mentioned above.
- DAOS base path: Specify a file path for DAOS storage. The default is "DAOS." This puts all NLOs under the /dominoserver/daos directory.
- Defer object deletion for (days): The default is 30. This is recommended.
5. Add "Create_R85_Databases=1" to the server's notes.ini file for a Domino 8.5 server. If you have Domino 9, add "Create_R9_Databases=1" instead.
6. Restart the server.
In order to update a set of NSF files to be DAOS-enabled, you need to do one of the following:
1. On the Advanced tab of the NSF's Application Properties box, select the "Use Domino Attachment and Object Service" check box.
2. Using the Domino Administrator, select a Notes database from the Files tab and choose Tools > Database > Advanced Properties; then select the "Use Domino Attachment and Object Service" option.
3. From the server console, issue a "load compact -daos on dbname" where dbname is the name of a Notes database. You can do a "load compact -daos on mail", which will operate on all NSF files under the mail directory.
Each option will also update the On-Disk Structure of any NSF to either 51 or 52, depending on the notes.ini Create_R85_Databases=1 or Create_R9_Databases=1 value you set.
Important: Option 3 is the only option to move existing eligible attachments into NLO files. Options 1 and 2 will only set the NSF to move any new eligible attachments into NLOs. So if you want to see the real value of DAOS, be sure to use option 3.
With option 3, the NSF will be compacted, turning on DAOS and moving its attachment data into NLO files. If Domino is a significant portion of your system storage, you can watch your disk activity and see the available storage increase. The processor overhead for DAOS is minimal and well worth it in terms of disk space savings.
On-Disk Structure (ODS)
In order to implement DAOS on a NSF file, its ODS must be at version 51 (Domino 8.5). Being able to implement DAOS is great, but the real value of ODS 51 is in the I/O savings. How much value? The difference between ODS 43 (Domino 7) and ODS 51 is up to 80% less file I/O! I'm not sure what the official I/O improvement is of Domino 9 (ODS 52) over Domino 8.5, but there have been improvements in I/O plus folder optimization, compression, and attachment consolidation.
I helped a customer that recently jumped from Domino 6.5 to Domino 8.5.3. A few days after the upgrade, I noticed most of their ODS levels were all still at version 20 (circa Domino 4). This was a long-time Domino customer, but they never updated ODS when they updated their Domino versions. After updating to ODS 51, their performance went through the roof. Scheduled agents completed in record time.
So even if you're not looking to implement DAOS, you'll need to update the ODS of your NSF files to ensure your servers are functioning to peak performance.
In order to update the ODS for all databases—including NSFs like the public address book (names.nsf) that are always in use by the server—you'll need to run the following command with the Domino server in a *ENDED status:
RUNDOMCMD SERVER(servername) CMD(CALL PGM(COMPACT) PARM('-ODS' '-*')) BATCH(*YES)
This will ensure that all files on the selected Domino server—including Notes Template Files (NTF) and any mail.box files—are on the latest ODS. The -ODS switch on the compact command will ensure that only files that aren't on the latest ODS will be upgraded so it's not spinning its wheels compacting already-updated files unnecessarily.
You can watch the progress of this command via the Domino console on a 5250 session. Once finished, it will tell you how many files were at an older ODS and how many were converted to the new ODS. If you have any issues with updating an ODS, it's usually an indicator of database corruption of some degree. Running standard database maintenance tasks like fixup, updall, and compact should clear that up; then you can run the command again.
Domino databases have a number of different compression options under the Application Properties dialog box, all of which save space on disk and offer I/O improvements with very little processor overhead.
- Compress Document Data—This is deselected by default. When selected, it reduces the size on disk of all rich-text elements in documents, and savings can be significant, especially when a database contains much text and images.
- Compress Database Design—This is deselected by default. When selected, it reduces the size of a design note by 55-60% and also reduces I/O.
- LZ1 Compression for Attachments—LZ1 offers a more advanced form of attachment compression than the older Huffman algorithm. You need to be at least at ODS 43 to enable this feature.
Domino servers, by default, run out of the *BASE memory pool. If you have some memory contention on your partition, you are able to segregate your Domino workloads from each other or from other applications running in *BASE by creating additional memory pools and assigning your Domino servers to them. Since minimum and maximum memory levels are set in each pool, you can ensure that production Domino is guaranteed memory resources. You can also guarantee test Domino servers are insulated by having them run in a smaller, separate pool so that they won't impact memory performance on the rest of the partition. Moving out of *BASE gives you much more granularity to ensure that servers get the resources they need.