Filtering and Searching In Visible Content With AngularJS (Part 1/2)

With angular filters you can realize full text searches in small data sets with minimal efforts. This is at least what the documentation suggests. In real-world applications, a naïve application of filters to the data model usually leads to more or less arbitrary behavior.

Let's look at an appliation where it works, the notorious Angular phonecat app. Searching for "Moto" in the search field reduces the list to entries with Motorola devices.

The cool thing about it is that the functionality is almost completely realized in the HTML view code:

<div class="container-fluid">
  <div class="row">
    <div class="col-md-2">
      <!--Sidebar content-->

      Search: <input ng-model="query">

    </div>
    <div class="col-md-10">
      <!--Body content-->

      <ul class="phones">
        <li ng-repeat="phone in phones | filter:query">
          {{phone.name}}
          <p>{{phone.snippet}}</p>
        </li>
      </ul>

    </div>
  </div>
</div>

In line 6, the value of the search input field is bound to the variable query, and in line 13 the repeater gets filtered with the contents of that variable.

I have prepared an application that looks very similar at first glance, a subscriber list of a fictitious forum:

The list contains the first and last name of all subscribers, the email address (or a hyphen '-' for those subscribers who objected to making their email address public), the subscription date, the number of postings and the country of origin. The user can hide the country column by checking the corresponding check box.

We want to list all subscribers from Italy:

After entering an "I" into the search field, the visible part of the list did not change but you can see the effect of the filter next to the search field. The label there informs you that only 172 of 198 entries are being displayed now. It is not unusual that so many lines contain an "I", and the result is therefore not a surprise.

But a thorough check reveals that the entry for the subscriber "Frank Gardner" actually does not contain any "I", neither upper or lower case.

After entering the next letter "t" we see the list filtered for "It":

Only 37 of 198 entries are left but our pal Frank Gardner is still listed.

Continuing the search for "Ita" shrinks the list to one lone entry:

The i-less Frank Gardner has disappeared. The only subscriber left is Saro Napolitano from Italy but all his fellow countrymen have vanished. Finally, after entering one more character and filtering for "Ital" the list is empty:

We would expect to see at least all entries containg "Italy" but no entry at all passes the filter. We must have a blunder in our logic. In order to understand the bug we have to dig a little deeper into the source code of the application.

Anatomy Of the Example Application

You can also follow the example and the following steps locally:

$ git clone -b broken git://git.guido-flohr.net/web/angular/angular-filter-visible-content.git
$ cd angular-filter-visible-content
$ npm start

Provided that "git" and "npm" are in your path, you can see the example now at http://localhost:8000/app/.

The actual application in app/app.js is very simple:

'use strict';

angular.module('myApp', []);

Following is the fictitious subscriber database:

angular.module.constant('SUBSCRIBERS', {
   {
      "country" : "IT",
      "surname" : "Pirozzi",
      "id" : "jzhodg-6388-694720",
      "email" : "a.pirozzi@costa.it",
      "postings" : 3969,
      "givenName" : "Albano",
      "showEmail" : true,
      "date" : 1246609342000
   },
   {
      "surname" : "Jankowski",
      "email" : "i.jankowski@zalewski.pl",
      "id" : "egkyan-6955-444173",
      "country" : "PL",
      "givenName" : "Iwan",
      "postings" : 4482,
      "showEmail" : true,
      "date" : 1246743355000
   },
   // ...
});

The problem with the filtering respectively search becomes clear now. The raw data does not contain the full country name (e. g. "ITALY") at all but only the ISO-3166-1 code for it. The search string "It" matches all entries with "IT" but "Ita" no longer matches. The only remaining hit is purely incidental because Saro Napolitano contains the search string in the last name.

The entry for Frank Gardner looks like this:

{
      "date" : 1319694218000,
      "showEmail" : false,
      "givenName" : "Frank",
      "postings" : 4684,
      "id" : "vfydee-7981-686357",
      "email" : "f.gardner@nesbitt.net",
      "surname" : "Gardner",
      "country" : "UK"
   },

This reveals another problem: Frank Gardner's entry matched on "It" because the mail address "f.gardner@nesbit.net" contained the search string. However, the flag "showEmail" is set to false so that the mail address is suppressed. But it still contributes to the filtered data and can produce hits.

Before considering possible solutions to the problem we have to get a more complete idea of the source code.

The controller app/components/subscribersController.js sorts the data hash and exports in a sorted array into the $scope variable subscribers:

'use strict';

angular.module('myApp')
.controller('subscribersController', [
    '$scope',
    'SUBSCRIBERS',
function($scope, SUBSCRIBERS) {
    $scope.subscribers = SUBSCRIBERS.sort(function(a, b) {
        if (a.postings < b.postings)
            return +1;
        if (a.postings > b.postings)
            return -1;
        return 0;
    });
    $scope.showCountry = true;
}]);

The data hash gets sorted by the number of postings in lines 8 to 14. The flag showCountry triggers the visibility of the country column, an admittedly rather bogus feature.

The relevant part of the view app/index.html is pretty standard for AngularJS:

<div class="row">
  <div class="col-md-8">
    <label for="query">
      Search:
    </label>
    <input name="query" ng-model="query" id="query"
           placeholder="Enter query!" class="search" autofocus>
  </div>
</div>
  <div class="col-md-7">
    <em>
      Displaying {{ (subscribers | filter: query).length }}
      of {{ subscribers.length }} entries.
    </em>
  </div>
  <div class="col-md-2">
    <input type="checkbox" ng-model="showCountry" id="show-country"
           ng-true-value="true" ng-false-value="false">
    <label for="show-country">
      Show country
    </label>
  </div>
<!-- Table header omitted. -->
<div class="row table-row"
     ng-class="{'odd': $index % 2 === 1, 'even': $index % 2 === 0}"
     ng-repeat="subscriber in subscribers | filter: query">
  <div class="col-md-3">
    {{ subscriber.givenName }} {{ subscriber.surname }}
  </div>
  <div class="col-md-3">
    <span ng-show="subscriber.showEmail">{{ subscriber.email }}</span>
    <span ng-hide="subscriber.showEmail">-</span>
  </div>
  <div class="col-md-2">
    {{ subscriber.date | date }}
  </div>
  <div class="col-md-2">
    {{ subscriber.postings | number }}
  </div>
  <div class="col-md-2" ng-show="showCountry">
    {{ subscriber.country | isoCountry }}
  </div>
</div>

The repeater is defined in line 26. The individual entries are stored in the loop variable subscriber. The current filter (search query) is stored in the variable query that is bound to value of the input field in line 6.

The name (line 28) and the email address (line 31) are passed through as is, all other fields are filtered by Angular directives. The subscription date is stored as milliseconds since the epoch (January 1st, 1970, 00:00 UTC) and is transformed into a human readable form with the directive date.

The filter number (line 38) also belongs to the AngularJS core and formats numbers into a locale-specific format, here the US American format with a comma as thousands separator.

Line 49 show the usage of the filter isoCountry from the package iso-3166-country-codes-angular that generates English country names from ISO-3166-1 country codes.

Problem Analysis

Three cases have to be considered.

A table column may display a field from the data model as is. This is the case for the email address in the example application. Really? No! The subscriber may opt out of publishing of the mail address. In that case it will not be displayed.

The column with the subscriber's name does also not fit completely because it contains the first and the last name separated by a space. Searching for "Saro" and also searching for "Napolitano" both trigger "Saro Napolitano" as a hit but searching for the full name "Saro Napolitano" does not because the first and the last name are stored in separate fields in the data model.

The second case is that the displayed value does not exist in the data model or has been generated by a transformation. For example, this is the case with the country column or the column with the subscription date.

Finally there are fields in the data model that are ignored for display, for example the subscriber id. These fields can lead to false positives because their contents is evaluated for filtering.

Possible Solutions

The data could be cleaned in the controller and be passed completely processed to the view components. However, that means that Angular directives must not be used at all inside of Angular repeaters. That approach conflicts with the MVC paradigm because view and controller cannot be cleanly separated. For example, you have to format numbers in the controller or translate strings, both things that are considered strong indicators of a bogus design.

You also have to take into account that this work has to be done for every single case. No way that you can generalize it. And a GUI that reacts to user interaction voids this approach altogether. The example application allows showing and hiding the country column. If you want to mirror that functionality inside of the controller you will find yourself writing an onclick hander that copies data back and forth from the data model into and out of the view.

That does not look like a viable approach.

Can we solve the problem by modifying the invocation of the filter? The syntax looks like this:

{{ filter_expression | filter : expression : comparator }}

expression can be a simple string, a JavaScript object or a function.

Passing an object allows you to realize complex rules by specifying conditions for multiple properties at once. That does not help here.

You can also pass a custom function instead. This lets you realize arbitrary filters. Says the AngularJS documentation. The function gets invoked for every single row in the data model and can include or exclude each row by returning true oder false. At first glance, that looks like the solution to our problem but it is not:

• The callback function has to "know" exactly what is going on in the view, more precisely, which transformations are applied.

• The callback function has to be written individually for every single use case. It cannot be generalized.

At the end of the day we repeat view code in the controller. This is not only ugly but also does not help because it voids the ngFilter functionality almost entirely.

The last optional argument for an Angular filter is the comparator. You can specify a custom comparison function that compares the user supplied search string against the result of the various transformations of the raw input data. But we run into the same problems: The function has to be adopted to the particular view code and we blur the distinction between view and controller, we create a solution that is not only ugly but also hard to maintain and error-prone.

Hint: If you do prefer one of the ugly solutions outlined above be prepared for unexpected detail problems. The solutions are not only ugly but also complicated!

User Expectation

Let's stand inside our users' shoes for a second. We would not know anything at all about hidden input fields or hidden content that only gets displayed under certain circumstances. We also would not know about auxiliary variables or garbage contained in the data model.

We just want to search what we can see. And that is exactly what a solution has to provide. We do not want to search the underlying raw input data but only what the individual user actually sees in particular situation.

Conclusion

Filters in AngularJS are a lot less useful than they look at first glance. They work very well for address books or CD databases, the kind of applications that we know from tutorials for relational data models. As soon as the data comes from real sources we run into problems because filters are applied to the raw input data and not to the displayed data derived from it.

In the next part we will develop a solution that tackles the problem in a generic manner.