Domino JNA Virtual Views: The Next Step in Domino Data Retrieval
Karsten Lehmann July 14 2024 12:10:00 AM
In the previous two articles, "The pain of reading data as a Domino developer - and solutions" and "Overview of Domino Data Retrieval: Exploring NSF Search, DQL, Domino Views, and the QueryResultsProcessor", we explored the challenges and solutions for efficiently accessing and processing data in Domino.Now, let's delve into the next evolution in Domino data handling.
Version 0.9.49 of our Domino JNA open-source project introduced a new API called "Virtual Views", built with all the NIF, formula search, DQL, and QueryResultsProcessor history and limitations in mind.
In short, you define the view structure as a VirtualView object in your Java code (columns with sorting, categorization, total, and average values) and then add one or more data providers that incrementally push data into the view (or remove it).
This article will guide you through the innovative features and capabilities of the Virtual View API, demonstrating how it simplifies and enhances data retrieval and manipulation in Domino.
Now, let's take a closer look at how you can leverage the Virtual View API to define and manage views dynamically in your Java code.
To illustrate, consider the following code:
someExternalData.put("Revoco", "Revoco Street 5, Los Angeles");
someExternalData.put("Omnis", "Omnis Boulevard 12, New York");
// by using "createViewOnce", we mark the view to be stored in memory,
// as a version "1" and to auto discard it
// after 5 minute of inactivity (just for testing, in production you'd use a higher value)
// changing the version number to "2" would force a new view to be created
VirtualView view = VirtualViewFactory.INSTANCE.createViewOnce("fakenames_namelengths",
1, // version "1"
5, TimeUnit.MINUTES, // auto discard after 5 minute of inactivity
// (calling createViewOnce resets the counting)
(id) -> {
return VirtualViewFactory.createView(
new VirtualViewColumn("Lastname", "Lastname",
Category.YES, Hidden.NO, ColumnSort.ASCENDING, Total.NONE,
"Lastname"),
new VirtualViewColumn("Firstname", "Firstname",
Category.NO, Hidden.NO, ColumnSort.ASCENDING, Total.NONE,
"Firstname"),
new VirtualViewColumn("Total Name Length", "TotalNameLength",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.SUM,
new VirtualViewColumnValueFunction< Integer >(1) {
// this 1 is a version number for
// the column function, might become
// relevant later when we store the index to disk
@Override
public Integer getValue(String origin, String itemName,
INoteSummary columnValues) {
return columnValues.getAsString("Firstname", "").length() + 1 +
columnValues.getAsString("Lastname", "").length();
}
}),
new VirtualViewColumn("Average Name Length", "AverageNameLength",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.AVERAGE,
new VirtualViewColumnValueFunction< Integer >(1) {
@Override
public Integer getValue(String origin, String itemName,
INoteSummary columnValues) {
return columnValues.getAsString("Firstname", "").length() + 1 +
columnValues.getAsString("Lastname", "").length();
}
}),
new VirtualViewColumn("Company Address", "CompanyAddress",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.NONE,
new VirtualViewColumnValueFunction< String >(1) {
@Override
public String getValue(String origin, String itemName,
INoteSummary columnValues) {
// poor man's JOIN :-)
// we fetch the company address from a map
// using the company name as key
String companyName = columnValues.getAsString("CompanyName", "");
return someExternalData.getOrDefault(companyName, "");
}
}),
new VirtualViewColumn("Last Update", "LastMod",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.NONE,
"LastMod"),
// required to have the CompanyName value available in the summary
// buffer so that the Java column function can use it
new VirtualViewColumn("Company Name", "CompanyName",
Category.NO, Hidden.YES, ColumnSort.NONE, Total.NONE,
"CompanyName")
)
// add one or more data providers with an origin id (here "myfakenames1")
.withDbSearch("myfakenames1",
"Server1/Mindoo", "fakenames.nsf",
"Form=\"Person\"")
// let's move the categories above the docs like in Windows Explorer
.withCategorizationStyle(CategorizationStyle.CATEGORY_THEN_DOCUMENT)
.build();
});
In this example, column values are computed via formula and Java code, and a JOIN operation with an external Java Map is performed.
More examples can be found here: https://github.com/klehmann/domino-jna/blob/develop/domino-jna/src/test/java/com/mindoo/domino/jna/test/TestVirtualView.java
The data provider implements a simple Java interface, making it easy to add non-Domino data to the view. For Domino, we built several standard implementations that can be mixed and matched for one or multiple databases:
- datasource 1: run NSF search with a formula (incrementally), can search data and design documents, optional post processing with FT search
- datasource 2: profile documents filtered with a formula
- datasource 3: read note ids from a folder (incrementally), ideal for displaying folder content in different ways
- datasource 4: compute column values from any list of note IDs (e.g. a DQL result)
The VirtualView class manages the categorization tree and document sort order, aggregating computation results on category levels up to an imaginary root element.
Readers lists
Readers lists are a first-class citizen in the Virtual View API. For documents, we store the individual readers within the view index entry, along with the origin database information. When traversing the view, we compare the readers list with the usernames list of the current user in that specific database. This also works with ACL roles which are defined per database.
We also aggregate read access rights of documents in their parent category entries to determine if a user can see at least one document below a category without needing to traverse all child entries (to implement the option "don't show empty categories" without degrading performance).
These aggregated readers stats can be read in code, offering insights into the distribution of read access rights across multiple databases.
Virtual Views do not need to be created on a per-user basis. The VirtualView object contains all data the server is allowed to see. To traverse the virtual view, we built our own VirtualViewNavigator, instantiated for a specific user and for the whole view or a single category:
VirtualViewNavigator nav = view .createViewNav()
.withCategories()
.withDocuments()
.withEffectiveUserName("CN=Karsten Lehmann/O=Mindoo")
.build();
// other methods to build the navigator:
// .buildFromCategory("Abbott");
// .buildFromDescendants(otherViewEntryInTheView);
nav.expandAll().collapse("1.1");
if (nav.gotoFirst()) {
do {
VirtualViewEntryData currentEntry = nav.getCurrentEntry();
String firstName = currentEntry.getAsString("firstname", "");
String lastName = currentEntry.getAsString("lastname", "");
}
while (nav.gotoNext());
}
gotoPrev() and gotoNext() manage the expanded/collapsed entries. If you prefer a non-goto syntax, methods like entriesForward(), entriesBackward() and others return Java Streams:
Stream stream = nav .entriesForward()
.skip(3000)
.limit(200);
stream
.forEach(((entry) -> {
String firstName = entry.getAsString("firstname","");
String lastName = entry.getAsString("firstname","");
}));
The view index is currently stored in memory. We may add a feature to save indexes to disk for quick reloading after an HTTP task restart.
Conclusion
Here is the full feature list for the Virtual View API in its first version:
- Multi-DB views
- Works locally, client-server and server-server
- View structure similar to Domino (multi-level categorization, sorted columns)
- Support for sums/average values
- Compute column values via formula or Java code
- Option to place categories above or below documents in categorized views
- Incremental view updates (no rebuild required)
- Full control over when the view is updated, with optional read locks for exclusive access
- Server-populated views, shared across users
- User-specific view entry visibility checks (DB ACL level and usernames list for each DB compared with computed document readers list)
- Aggregated readers stats for categories to quickly skip empty categories for users
- Collected readers stats for analysis purposes
- Several data providers combined to produce view data
- Standard Domino data providers for data/profile/design docs
- Custom non-Domino data support
- VirtualViewNavigator for reading view entries visible to a user, with support for expanded/selected entries, upwards/downwards paging, keyword, and range lookups
- Fast performance (e.g., processes 40,000 fake name docs and builds the view in 2-3 seconds)
- VirtualView currently stored in Java heap (each VirtualViewEntry with a ConcurrentSkipListMap for sorted children), with potential future support for disk serialization
- Not Domino-specific; works for other kinds of data
I hope you enjoy this new feature, try it out, and provide feedback!
- Comments [0]