Asynchronous data providing

To view the complete code for the examples described in this document, download the AsynchronousDataLoading project.

The ListView in Cascades is designed to be able to display large amounts of data while still maintaining good performance. To achieve this performance, only a small subset of list items are held in memory at any given time (those that are currently within the visible area and a handful above and below). This means that whether a ListView has 10 items or 2000 items in its DataModel, it will have the same performance.

When a user scrolls the list, the ListView requests new data from the DataModel in the direction that the list is scrolling. Instead of creating new list items for the data, the ListView reuses list items that have been scrolled out of the visible area of the screen.

For the list to perform well, it needs to be able to retrieve and process new data very quickly. These are some of the functions that are called while a ListView is scrolling:

Diagram showing scrolling in a list view.

Creates a VisualNode for each visible list item when the ListView is initialized.

Updates the list item based on the provided type, index path, and data. This function is called whenever a new item is about to be displayed in the list. Whether or not this function returns quickly enough does not affect the performance of the list, but it does affect whether the correct data is displayed in the list item.

Returns the number of children of a list item specified by an indexPath. This value is used by the ListView to display a scroll bar that's of an appropriate size for the list. This function is always called if you implement your own DataModel.

Returns the item type of the list item, based on the data that it contains. This function is only called if a ListItemTypeMapper is registered on the ListView.

For the list to be able to scroll quickly and load data right away, you must ensure that your DataModel is ready to respond immediately to the calls mentioned above. The best way of doing this is to have your DataModel act as a cache that pre-fetches data asynchronously so that data that is always available to the ListView. Asynchronous database requests are handled using the SqlConnection class.

Asynchronous data providing for lists

Depending on your application, there are a few different ways that you might want to construct your DataModel. Here are two possible solutions that you can use.

Solution #1

In the first solution, the DataModel only reports the items already loaded into its internal data structure (the amount of data in the DataModel is specified by DataModel::childCount()). When the list gets closer to the end of this internal data structure, you simply add more items to the edge of the internal data structure in the direction the list is being scrolled. Once new items are added to the model's internal data structure, it emits the itemAdded() signal. When the internal data structure of the DataModel gets too large, you simply remove items from the side of the data structure in the opposite direction that the list is being scrolled. When items are removed, the model emits the itemRemoved() signal.

The advantage of this solution is that the list stops scrolling when there aren't any items loaded in the cache, allowing you to show an ActivityIndicator at the bottom of the list. The drawback to this solution is that the scroll bar for the list jumps when new items are added and removed. The solution is mostly suited for online data since this type of scroll bar behavior is normal in those cases.

Solution #2

In the second solution, list items are never fully removed from the internal structure of the DataModel when it gets too large. Even though the data itself is thrown away, the DataModel always retains the database ID of the items so that a list item's indexPath in the model always corresponds to an item in the database. One of the benefits of this solution is that the scroll bar always appears at an accurate position in the list. One of the drawbacks is that some list items might not display their data initially if loading does not happen quickly enough.

Here are the steps for how to populate the model using this solution.

  1. First, you create the ListView with an empty DataModel and optionally start an ActivityIndicator so that the user knows the application is working.
  2. Next, you load a small set of data from the data source and store it in the internal data structure of the DataModel.
  3. Once the initial set of data loads, you asynchronously load a larger set of data.
  4. With the initial set of data in place, the DataModel reports on the number of items in its internal data structure to the ListView using DataModel::childCount().
  5. While the list is scrolling, the ListView calls ListItemProvider::updateItem() to update list items with data before they are displayed. If the index path of the list item is approaching the end of what's available in the internal structure, you must asynchronously load more data into the internal data structure.
  6. When the internal data structure gets too large you must remove data from the internal structure on the end furthest from the current scroll position of the ListView (the scroll position can be estimated using ListItemProvider::updateItem()). The database IDs for the list items must not be removed as they are required to track the correct positions of list items in the DataModel.

Implementation

Let's have a closer look at how solution #2 is implemented. The first step is to create the DataModel. In addition to overriding some of the standard DataModel functions, we also create functions for loading and removing items from the DataModel, and for requesting/receiving data from a SQL data source.

class MySqlDataModel: public DataModel {
Q_OBJECT

public:
	MySqlDataModel();
	virtual ~MySqlDataModel();

	// Implementing the DataModel class
	Q_INVOKABLE
	int childCount(const QVariantList &indexPath);Q_INVOKABLE
	bool hasChildren(const QVariantList &indexPath);Q_INVOKABLE
	QVariant data(const QVariantList &indexPath);

	// Functions for loading data and removing items
	Q_INVOKABLE
	void removeItems();Q_INVOKABLE
	void loadData();

	// A signal that's emitted when the initial set of data has been
	// loaded into the DataModel
	Q_SIGNAL
	void initialItemsLoaded();

	// Signals and slots used for initiating the SQL request
	Q_SIGNAL
	void loadSQLData(const QString&);Q_SIGNAL
	Q_SLOT
	void onLoadData(const QString& sql);Q_SLOT

	// A slot that asynchronously receives the reply data from the 
	// SQL request
public Q_SLOTS:
	void onReply(const bb::data::DataAccessReply& dar);

private:
	void removeLoadingItem();

	SqlWorker *m_pSqlWorker;
	SqlConnection *m_pSqlConnection;

	QVariantList modelList;
	int mState;
};

The next step is to implement an IndexMapper, which allows the DataModel to maintain the correct indices for list items when data is added and removed.

#ifndef MYINDEXMAPPER_H_
#define MYINDEXMAPPER_H_

#include <bb/cascades/DataModel>

class MyIndexMapper: public bb::cascades::DataModel::IndexMapper {
public:
	MyIndexMapper(int index, int count, bool deleted);
	bool newIndexPath(QVariantList *pNewIndexPath, int *pReplacementIndex,
			const QVariantList &oldIndexPath) const;

	int mIndex;
	int mCount;
	bool mDeleted;
};

#endif /* MYINDEXMAPPER_H_ */

IndexMapper contains a function called newIndexPath(), which is invoked for every item in a data model's internal data structure, each time the itemsChanged() signal is emitted (this happens every time an item is added to or removed from the DataModel). In our implementation of IndexMapper, we need to be able to handle three different scenarios when itemsChanged() is emitted.

Items with an unchanged index

Generally, these are items with a lower index than the added or removed items. No action needs to be taken for these items.

Items with a changed index

Generally, these are items with a higher index than the added or removed items. The index path for these items must be readjusted depending on how many items were added or removed.

Deleted items

For these items, you return the index that the deleted item would have had so that the items are removed from the correct location in the ListView. If a number of sequential items are removed at the same time, the same index is returned for each one, provided that they are children to the same parent.

bool MyIndexMapper::newIndexPath(QVariantList *pNewIndexPath,
		int *pReplacementIndex, const QVariantList &oldIndexPath) const {

	// Unaffected indices
	if (oldIndexPath[0].toInt() < mIndex) {
		(*pNewIndexPath).append(oldIndexPath);
		return true;

	// Deleted indices
	} else if (mDeleted && oldIndexPath[0].toInt() <= mIndex + mCount) {
		*pReplacementIndex = mIndex;
		return false;

	// Indices after a deletion or addition
	} else {
		if (mDeleted)
			(*pNewIndexPath).append(oldIndexPath[0].toInt() - mCount);
		else
			(*pNewIndexPath).append(oldIndexPath[0].toInt() + mCount);
		return true;
	}
}

The final step is implementing the asynchronous requests for SQL data. The loadData() function shows how the SQL data is gradually obtained over a few steps. First, a very small set of data is requested, followed by a larger set of data, and then the remainder of the data.

void MySqlDataModel::loadData() {
	switch (mState) {
	case 0:
		emit loadSQLData("select * from contact limit 20");
		break;
	case 1:
		emit loadSQLData("select * from contact limit 30 offset 20");
		break;
	case 2:
		emit loadSQLData("select * from contact limit 50000 offset 50");
		break;
	default:
		return;
	};
	mState++;
}

To asynchronously request the data, you create a SqlConnection object and connect its reply() signal to a slot that you create to handle the reply (onReply()). You start the request by calling execute().

void MySqlDataModel::onLoadData(const QString& sql) {
	if (!m_pSqlWorker) {
		m_pSqlWorker = new SqlWorker("app/native/assets/sql/contacts1k.db",
				this);
		m_pSqlConnection = new SqlConnection(m_pSqlWorker, this);		
		
		// If any Q_ASSERT statement(s) indicate that the slot failed to connect to 
		// the signal, make sure you know exactly why this has happened. This is not
		// normal, and will cause your app to stop working!!
		bool res;
		
		// Since the variable is not used in the app, this is added to avoid a 
		// compiler warning.
		Q_UNUSED(res);
		
		res =  connect(m_pSqlConnection,
				       SIGNAL(reply(const bb::data::DataAccessReply&)), 
				       this,
				       SLOT(onReply(const bb::data::DataAccessReply&)));
				
		// This is only available in Debug builds.
		Q_ASSERT(res);
	}

	m_pSqlConnection->execute(sql);
}

Once you receive the reply data asynchronously, you simply need to append the data to the DataModel and inform the application that new data has been added.

Last modified: 2013-12-21

comments powered by Disqus