Bayanat is the second generation of SJAC’s open-source web data management application. SJAC had been using the first generation, Corroborator, since 2014. In late 2019, SJAC decided to rewrite the application from scratch using the Flask Python microframework. SJAC used its own six years of first-hand experience in collecting, preserving, processing, and sharing documentation of human rights violations to build Bayanat.
The main purpose of Bayanat is to attack the challenge of big data in human rights documentation head-on. SJAC has developed an analytical methodology based on international humanitarian and human rights law. Events are cataloged according to specific IHL violations. This allows the most relevant evidence to be identified at a keystroke, enabling a variety of transitional justice efforts. Despite its main focus on human rights documentation, Bayanat can be utilized to research any other topic.
Bayanat employs the industry’s best practices: safe HTML generation, sanitized inputs, hashed/salted passwords, encrypted communication, etc. User permissions are applied on both the front- and the back-end.
SJAC is also planning to introduce features to allow the database to effectively store forensic data and information on missing persons, including searchable fields for ante and postmortem data (necessary for missing persons projects, from Syria to Iraq to Yemen).
Bayanat has been beta-tested by established data management teams in human rights organizations and UN entities and has already been deployed in production for SJAC’s Data Analysis team’s work.
There are three main components of the Bayanat database:
A Bulletin is a single piece of documentation that provides information. Bulletins could be created from almost any type of content, such as videos, images or pictures, physical documents (or scans), interviews, reports, text files, audio files, etc. These pieces of documentation are imported to the database to create Bulletins. To complete Bulletins, the piece of documentation is analyzed, which involves processing by an analyst.
An Actor is an item of documentation about a person or entity that plays a role in an event, and/or is related to other Actors, Bulletins, or Incidents. These may include alleged perpetrators, witnesses, injured parties, or other categories. Information on Actors is collected from a variety of sources, such as organizations working to collect names of victims, sanctions lists, companies registers, etc. Actors may also be created by the user as they analyze Bulletins.
Incidents are where Actors and Bulletins related to the same event are bundled and analyzed in a detailed manner. They don’t have information themselves but combine information from other entities. This may relate to a particular event or course of events.
The other components of the database are:
Tiny form to add time and location information to the main components.
Profiles of all the sources attached to the main components.
List of locations in the country that can be added in different capacities to the main components.
Tags that can be added to the three main components. The visibility of labels in each of the three components can be controlled.
Each item has an “owner” (assigned user), who is the user responsible for processing the item, and the only user, except for an administrator, who can edit the item. The item can have a peer reviewer, who can review the work the assigned user has done and leave a review for them about any mistakes in the analysis, without editing the item itself.
The database has a simple workflow as well and only an administrator can move an item out of its workflow. Bulk operations are available to the administrators in order to assign items to users for analysis or review.
The app has four built-in roles: administrators, moderators, data analysts (DAs), and view-only users. DAs and moderators can only edit or review items assigned to them, and moderators can additionally create or edit Locations, Sources, Labels, and Event types. View only users have the ability to view and search items in the three main components.
Additional permissions to restrict the ability to view the usernames or the history of each item are available and can be switched on or off for each user.
Each time an entity in the database is edited, a snapshot of it is taken and saved. This allows the users with the appropriate permissions to view each item’s simple and detailed history. The simple history shows the user who performed the update, the time of the update, and the comment they left. The detailed history shows the difference between a version and the previous one. This “database” of snapshots can in the future be used for an “undo” feature where an item can be restored to a previous version.
Each login, logout, or any update in the database by all users are tracked and can be viewed by an administrator from the Activity Monitor dashboard.
The application has a powerful search and filter tool that allows users to easily build simple and complex queries, combine multiple queries together, and save queries and reuse them quickly.
Users have the ability to search specifically in each field of the three components and exclude any results, ranges can be used to search date fields.
The app employs the industry’s best practices; safe HTML generation, sanitized inputs, hashed/salted passwords, encrypted communication, etc. User permissions are applied on both the front and backend. The app also has two-factor authentication and can be configured to utilize Google OAuth 2.0 (sign in with Google accounts).
Well documented source code to make it easier for others to modify the app to their needs, clear and easy setup instructions.
Scripts to load and update data can be written with ease. SJAC utilized this to create new scripts to load new video Bulletins to the database as well as migrating data from Corroborator.
Clean and consistent user interface with light and dark modes.
The app will be shipped with English and Arabic locales, new locales are easy to create.