System Administration
=====================

Upgrading Critic
----------------

To install a newer version of Critic, simply check out a newer commit in the
<code>critic.git</code> repository clone from which Critic was installed, and
then run the command

| python upgrade.py

as root.  If the repository clone from which Critic was installed has been
deleted, upgrading Critic from a newly cloned repository will usually work as
well.  The only case in which this does not work is if the commit that was
installed (or last upgraded to) is not available in the newly cloned repository.

Upgrading Actions
-----------------
The actions taken by the <code>upgrade.py</code> are roughly these, in this
order:

1 Offer to create a backup copy (dump) of Critic's database,
2 install additional prerequisite software the the new version depends on,
3 ask questions for new configuration settings in the new version,
4 copy updated source files (to
    <code>%(configuration.paths.INSTALL_DIR)s/</code>),
5 write updated configuration files (to
    <code>%(configuration.paths.CONFIG_DIR)s/configuration/</code>)
  and other system files (e.g. Apache site definition file, SysV init script,
  <code>/usr/bin/criticctl</code>),
6 restart Apache and Critic background services, and
7 run migration scripts to update the database schema or perform other one-time
  system updating tasks.

Each of these steps could be empty, and thus skipped, depending on the changes
between the currently installed version of Critic and the new version.

Locally Modified Files
----------------------
If the upgrade process wants to write an updated version of an installed file,
it will first check if the currently installed file has been modified since it
was installed.  No local modification are reverted without asking first.  Note
though that modified configuration files can be modified without asking if this
can be done in a way that does not conflict with, and that preserves, the local
modifications.

Migration Scripts
-----------------
Some changes to the system requires custom modifications during an upgrade to
adjust the installed system to match how the new version of Critic would have
set the system up if it had been installed from scratch.  The most common such
change is changes to the database schema, such as adding new tables or new
columns to existing tables.

This type of custom modifications are performed by running small one-time
scripts called migration scripts.  When upgrading to a new version of Critic,
all migration scripts that exist in the new version that didn't exist in the
previously installed version are executed as the final step of the upgrade.

If the upgrade process would run any migration script that modifies the database
schema, the offer to create a backup copy of Critic's database at the beginning
of the upgrade process has "yes" as its default choice instead of "no" as it
otherwise has.  It is strongly recommended that the offer is accepted, since the
changes made by such a migration script are generally not reversible in any
other than to restore an earlier backup copy of the database.

Change Host Web Server
----------------------
Critic's installation/upgrading scripts support a few different host web server
configurations:

* Apache 2.x (with mod_wsgi)
* nginx (HTTP front-end) + uWSGI (WSGI back-end)
* uWSGI (as both HTTP front-end and WSGI back-end)

The Apache configuration used to be the only supported, so an old Critic system
probably runs that one.  The nginx+uWSGI configuration is the currently
preferred (and default) one, but the installation script will ask which one to
use when a new system is installed.

The <code>upgrade.py</code> script can be used switch to a different
configuration, i.e. replicate how a new install would have configured the
desired new web server configuration.  Doing this requires just a little bit of
manual administration work:

1 Stop the current host web server.  This is required so that it stops listening
  at the HTTP/HTTPS ports, so that the new web server can do so instead.  You
  can typically do that by running the command
    <code>service &lt;name&gt; stop</code>
  as root, where <i>name</i> is one of <code>apache2</code>, <code>nginx</code>
  and <code>uwsgi</code>.
2 Edit <code>%(configuration.paths.CONFIG_DIR)s/configuration/base.py</code>,
  changing the <code>WEB_SERVER_INTEGRATION</code> setting to the desired value.
  See the inline documentation in the file for details.
3 Run <code>python upgrade.py</code> as root.  Note that unless you're also
  upgrading Critic to a new version, you may need to add the
  <code>--force</code> command-line argument to force the upgrading script to
  run despite there being no changes to install.

The upgrading script will, if necessary, install the new web server, and
then install configuration files for it.

  Note that the script may warn that it failed to stop the <i>new</i> web server
  before the actual upgrade.  This is expected, since the new web server may not
  have been running, or even been installed, at that time.

After this, Critic should be set up and running in the new web server.  You may
need to make sure the previous web server doesn't start up again the next time
the system boots.  The easiest way to do that is probably to uninstall it
completely.

Configuring Critic
------------------

The installation and upgrading procedure will produce a fully configured Critic
system.  Not all details can be controlled, however, and sometimes it may be
necessary to change the installed configuration later on.  To do this, one needs
to edit the configuration files under
  <code>%(configuration.paths.CONFIG_DIR)s/</code>
and restart the host web server (to reload the web front-end) and Critic's
background services.  The best way to restart Critic is using the
  <code>criticctl</code>
utility:

| criticctl restart

Before restarting, it will run a basic configuration test to make sure the
configuration isn't obviously invalid.  This will not catch all possible errors,
but helps to prevent putting the Critic system in an unusable state.

It is also possible to just run the configuration test:

| criticctl configtest

Do note that while it is necessary to manually restart to immediately make
changes effective, some background services spawns child processes to do the
actual work, and they will pick up changes right away, without an explicit
restart.  Likewise, the host web server may spawn additional web front-end child
processes, or stop and restart existing ones, thus picking up changes before
being explicitly restarted.  Due to this fact, it's best to explicitly restart
immediately after making a change, so that any problems are detected and can be
addressed.

Git Repository Access
---------------------

Critic's Git repositories should normally be accessible to the users of the
system, both to clone and examine locally and to push new commits to.  It is not
strictly required that they are -- Critic can be set up such that users only
access Git repositories on some other system from which Critic in turn fetches
-- but it is the typical set-up.

Git supports a variety of schemes (or protocols) for accessing repositories.  In
general, any access scheme will work with Critic as long as doesn't involve
directly manipulating the repository on the Critic server in a way that bypasses
Critic's <code>pre-receive</code> hook.  Some access schemes require the system
administrator to perform administration tasks beyond installing and managing
Critic, however.

git://host/path.git
-------------------
The Git scheme, provided by the <code>git daemon</code> command, is typically
used only for read-only access to Git repositories.  It can support push as well
but it will be unauthenticated, which means there will be no way of knowing
which user is responsible for the push.  Critic has no support for starting and
running <code>git daemon</code>.

This simple command can be used to start a <code>git daemon</code> process that
provides read-only access to all of Critic's repositories on
%(configuration.base.HOSTNAME)s:

| git daemon --export-all --base-path=%(configuration.paths.GIT_DIR)s %(configuration.paths.GIT_DIR)s

The command needs to be executed as the Critic system user
(<code>%(configuration.base.SYSTEM_USER_NAME)s</code>), for instance using
<code>sudo</code>:

| sudo -H -u critic git daemon ...

For more details about how to run <code>git daemon</code>, see its built-in
documentation:

| git daemon --help

http://host/path.git
--------------------
The HTTP (and/or HTTPS) protocol, provided by the <code>git http-backend</code>
command, is supported by Critic and enabled by default.  If the Critic web
front-end allows anonymous users, then it also allows unauthenticated read-only
access.  Push is only supported from authenticated users.

ssh://host%(configuration.paths.GIT_DIR)s/path.git / host:%(configuration.paths.GIT_DIR)s/path.git
---------------------------------------------------------------------
Access to repositories over SSH simply requires users to be able to log into the
Critic system over SSH and be members of the
  <code>%(configuration.base.SYSTEM_GROUP_NAME)s</code>
system group; Git takes care of the rest.  See also the note on
  <a href="#system_users">System Users</a>
below.

  Hint: To allow users to access Git repositories over SSH without giving them
  "shell access," set the system users' shell to <code>git-shell</code>.

Configuration
-------------
There are two settings in Critic relating to repository access.

First, there's the system configuration setting
<code>REPOSITORY_URL_TYPES</code>, set in
<code>%(configuration.paths.CONFIG_DIR)s/configuration/base.py</code>, which is
a list of access schemes that are supported on the system.  If this list does
not contain the string <code>"http"</code>, the built-in support for this access
scheme is disabled.  The other schemes -- <code>"git"</code>, <code>"ssh"</code>
and <code>"host"</code> -- are not in themselves affected by the setting.

Second, there's the user preference setting CONFIG(repository.urlType) that
determines what type of repository URLs the web front-end displays.  Each user
can select their preferred URL type among the set of supported schemes as
defined by the system configuration setting above.

Users
-----

Critic has a user database containing records of the users of the system.  The
record for each user contains the (typically short) user name, the user's full
name (which is what is normally displayed) and the user's email address, which
is used when the system sends notifications to the user.

The user database is populated in different ways depending on how Critic was
configured to identify and authenticate users.

Authentication mode: Critic
---------------------------
If Critic was configured to authenticate users itself, then users can be added
to the database manually by a system administrator.  This can be done using the
  <code>criticctl</code>
utility (installed as
  <code>/usr/bin/criticctl</code>
by Critic's installation script):

| criticctl adduser --name=USERNAME --email=EMAIL --fullname=FULLNAME

  Note: The program will prompt for the user's password.  If any of the options
  are omitted, it will also prompt for that information.

It is also possible to let new users create user records in Critic themselves,
by changing the system setting
  <code>ALLOW_USER_REGISTRATION</code>
in
  <code>%(configuration.paths.CONFIG_DIR)s/configuration/base.py</code>.
When enabled, a link is added on the sign-in page to a separate page where users
can input the necessary information to create a user record.

Is is also possible to let users sign in, and optionally create user records, by
authenticating via an external system, such as GitHub or Google, using the OAuth
2.0 protocol.  For more information about this mechanism, see the [tutorial on
external authentication][extauth].

[extauth]: /tutorial?item=external-authentication

Authentication mode: Host
-------------------------
If Critic was configured to let the host web server authenticate users, it
instead assumes that the
  <code>REMOTE_USER</code>
WSGI environment variable is set to the name of the accessing user, and
automatically creates a user record in its database if one doesn't already
exist.  In this scenario, there's typically no reason for the system
administrator to manually add users to Critic's database (but it's still
possible to do so.)  See
  <a href="/tutorial?item=customization#automatic_user_creation">the customization tutorial</a>
for details on this automatic user creation, and how it can be tweaked.

Anonymous Access
----------------
It is possible to let users access a Critic system without signing in, by
setting the system setting
  <code>ALLOW_ANONYMOUS_USER</code>
in
  <code>%(configuration.paths.CONFIG_DIR)s/configuration/base.py</code>
to True.

When Critic was configured to authenticate users itself (authentication mode
"critic") and use HTTP cookies to handle sessions (instead of HTTP
authentication) allowing anonymous access simply means the system won't redirect
to the "Sign in" page for most page accesses; instead there will be a "Sign in"
link in the paqe header.

When Critic was configured to let the host web server authenticate users, the
user is simply considered anonymous when the
  <code>REMOTE_USER</code>
WSGI environment variable is unset.  The host web server must be configured
appropriately for this to be a meaningful configuration, and there are typically
limited use-cases.  One possible use-case is to make parts of the web UI
available anonymously by configuring the host web server to require a signed in
user for some paths and not for others.

System Users
------------
Critic's user database doesn't necessarily have to match the system's user
database.  However, if access to the Git repositories is provided over SSH,
then, for the purpose of SSH authentication, there needs to be system users.
And in this case, for each system user who can push changes to the Git
repositories over SSH there must be a matching user in Critic's user database
with the same username.

Also note that system users that should be allowed to access the Git
repositories must be members of the
  <code>%(configuration.base.SYSTEM_GROUP_NAME)s</code>
system group.

Git Authors and Committers
--------------------------
Critic's user database doesn't necessarily contain records for every user
occurring as the author or committer of commits in the Git repositories, but
commits can be mapped to Critic users by matching the email addresses in the Git
commit meta-data.  Each Critic user can register any number of email addresses
that he or she uses or has used when creating Git commits.  By default, the
user's primary email address is used to map commits to the user, but the user is
in no way forced to use this particular email address in Git commits.

Currently, the only effect of mapping a commit's user references to a Critic
user is that a Critic user is never automatically assigned to review commits
that he/she is the author of.

Deleting Users
--------------
It is also possible to "delete" a user using <code>criticctl</code>:

| criticctl deluser --name=USERNAME

This doesn't actually delete the user record from the database, since that is
likely to be referenced from many places, depending on what the user has done in
the system.  Instead, what deleting a user does is

1 Marks the user as "retired", which prevents the system from acting as if the
  user is expected to be reviewing any changes in the future, and
2 deletes the user's password so that he or she can no longer sign in.

<strong>Important note:</strong> Deleting a user's password only prevents access
to the system if Critic handles user authentication.  If the web server handles
it, the user must primarily be disabled in whatever mechanism the web server
uses to authenticate users.  Critic will <b>not</b> disallow access in this
case.  What's more, in this case, if a deleted/retired user signs in, the user's
"retired" status is automatically reverted, thus completely undoing the effects
of "deleting" the user in the first place.

Roles
-----

User roles is a very basic access rights scheme, limiting which Critic users can
do a small number of things, including adding new repositories, modifying other
users' information, and adding news items.  Other than that, all users can
access all information in the system, modify all reviews and so on.

The available roles are:

? administrator
= A user with the <code>administrator</code> role can restart system background
  services via the
    <code><a href="/services">%(configuration.URL)s/services</a></code>
  page, edit other users' information via
    <code style='white-space: nowrap'>%(configuration.URL)s/home?user=&lt;name&gt;</code>
  and also enable/disable any tracked branch via
    <code><a href="/repositories">%(configuration.URL)s/repositories</a></code>.
  More exceptions are likely to be added in the future.

? repositories
= A user with the <code>repositories</code> role can add new repositories to the
  system via the
    <code><a href="/newrepository">%(configuration.URL)s/newrepository</a></code>
  page.

? newswriter
= A user with the <code>newswriter</code> role can write news items items that
  appear on the
    <code><a href="/news">%(configuration.URL)s/news</a></code>
  page.  The user can also edit existing items.  It is likely that news items
  will be also be added automatically when upgrading the system, to inform about
  significant changes.

? developer
= The <code>developer</code> role doesn't really give access to restricted
  functionality.  Instead it affects how unexpected errors -- such as uncaught
  Python exceptions -- are handled.  Normally, unexpected errors are presented
  to the user as just that, an "unexpected error," with a note that a message
  has been sent to the system administrator, and an email is sent to the system
  administrator with as much details as possible about the error.  But if the
  user triggering the error has the <code>developer</code> role, no email is
  sent to the system administrator (who is likely to be the same person in
  practice) and instead the error details -- typically a Python stack-trace --
  is displayed directly to the user.

Assigning Roles
---------------
To assign a role to a user, use the
  <code>criticctl</code>
utility:

| criticctl addrole --name=USERNAME --role=ROLE

and to unassign a role:

| criticctl delrole --name=USERNAME --role=ROLE

  Note: If the options are not provided, <code>criticctl</code> will instead
  prompt for the missing information.

Background Services
-------------------

The running Critic system is divided into two parts: the web front-end code,
running as a WSGI application (in Apache, using mod_wsgi) and a number of
background services running as daemon processes (started via a SysV init
script.)  The background services are listed on the page
  <code><a href="/services">%(configuration.URL)s/services</a></code>
where the system administrator (any user with the <code>administrator</code>
role) can restart them and view their logs.  The WSGI daemon processes can also
be restarted from that page.  By default there are two WSGI daemon processes;
this number can be adjusted by modifying the <code>WSGIDaemonProcess</code>
directive in the Apache site definition file.

Normally, the system administrator does not need to do anything to any of the
background services; they are restarted automatically when Critic is upgraded or
if any of them crash.  If any errors or warnings are logged by a background
service, an email is sent to the system administrator email address.

Background services are configured in the configuration file
  <code>%(configuration.paths.CONFIG_DIR)s/configuration/services.py</code>
where the set of services to start is defined, as well as various per-service
configuration settings.  If these settings are changed, the services need to be
restarted for the new settings to take effect.

The following background services are started automatically by Critic:

branchtracker
-------------
The <code>branchtracker</code> service is responsible for fetching tracked
branches from remote Git repositories and pushing them to Critic's repositories.
If a Critic repository is created to mirror another Git repository, one such
tracked branch is automatically set up, typically named <code>master</code>.
Additional tracked branches can be added via the
  <code><a href="/repositories">%(configuration.URL)s/repositories</a></code>
page by any user with the <code>administrator</code> role.  Reviews can also be
created to track branches in other Git repositories using the
  <code><a href="/createreview">%(configuration.URL)s/createreview</a></code>
page.

changeset
---------
The <code>changeset</code> service is responsible for processing diffs and
storing cached information about them in Critic's database.  The service also
periodically scans the database for cached diffs that haven't been accessed for
some time, and deletes them.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| CHANGESET["max_workers"] = 4
| CHANGESET["rss_limit"] = 1024 ** 3
| CHANGESET["purge_at"] = (2, 15)

The "max_workers" setting decides how many parallel jobs the service can run.
The "rss_limit" setting is a safe-guard against run-away memory usage.  The
"purge_at" setting is a tuple (HOUR, MINUTE) that defines when during the day
(in the server's local time) that the service should perform its maintenance
tasks.

githook
-------
The <code>githook</code> service performs all the checks and updates needed when
Critic's repositories are updated by users using <code>git push</code>.  The
actual <code>pre-receive</code> hook that is installed in Critic's repositories
is a simple script that connects to the service over a UNIX socket and forwards
all information about the update to the service.

highlight
---------
The <code>highlight</code> service is responsible for generating syntax
highlighted copies of source file versions and storing them in a cache, as well
as periodically compressing and eventually deleting old highlighted copies.

As part of syntax highlighting, "code contexts" are in some cases (depending on
the language of the source file) calculated and stored in Critic's database.  A
"code context" is a short string, for instance a function signature, mapped to a
range of lines, and used by the diff viewer to give context to the code
fragments it displays.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| HIGHLIGHT["cache_dir"] = os.path.join(configuration.paths.CACHE_DIR, "highlight")
| HIGHLIGHT["min_context_length"] = 5
| HIGHLIGHT["max_context_length"] = 256
| HIGHLIGHT["max_workers"] = 4
| HIGHLIGHT["compact_at"] = (3, 15)

The "cache_dir" setting defines where cached highlighted copies are stored on
disk.  The "min_context_length" and "max_context_length" limits length of code
context strings.  The "max_workers" setting defines how many parallel jobs the
service can run.  The "compact_at" setting is a tuple (HOUR, MINUTE) that
defines when during the day (in the server's local time) that the service should
perform its maintenance tasks.

maildelivery
------------
The <code>maildelivery</code> service is responsible for picking up queued mails
produced by other parts of Critic and delivering them to the configured SMTP
server.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| MAILDELIVERY["timeout"] = 10

The "timeout" setting defines the (socket) timeout used when contacting the SMTP
server.  If a connection or mail delivery attempt times out, the service will
wait a while and then try again.

maintenance
-----------
The <code>maintenance</code> service performs miscellaneous maintenance tasks
periodically.  Currently it performs these tasks:

* Keeping the UTC offsets in the <code>timezones</code> table in Critic's
  database up-to-date.  The offsets are fetched from PostgreSQL's
  <code>pg_timezone_names</code> table, but accessing this table directly is
  quite slow, so the offsets in it are read periodically and cached.
* Performing scheduled [review branch archivals][archival].
* Running <code>git gc</code> in Critic's Git repositories.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| MAINTENANCE["maintenance_at"] = (4, 0)

The "maintenance_at" setting is a tuple (HOUR, MINUTE) that defines when during
the day (in the server's local time) that the service should perform its
maintenance tasks.

[archival]: #review_branch_archival

servicemanager
--------------
The <code>servicemanager</code> service is a meta-service that starts all other
services, and restarts them if they crash.  If this service is restarted it
restarts all other services.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| SERVICEMANAGER["services"] = [...]

The "services" setting is a list of services to start.

watchdog
--------
The <code>watchdog</code> service is a simple monitoring service.

By default it only monitors the memory usage of the WSGI daemon processes and
restarts them if their memory usage exceeds the configured limits.  There are
two limits; a soft limit at which it tells the WSGI process to restart itself by
sending it a <code>SIGINT</code> signal, and a hard limit at which it kills the
WSGI process by sending it a <code>SIGKILL</code> signal.

If configured to do so, the service can also monitor overall system load and
send emails to the system administrator if the load exceeds configured limits.

The service has the following additional configuration settings in
<code>services.py</code> (with the default settings):

| WATCHDOG["rss_soft_limit"] = 1024 ** 3
| WATCHDOG["rss_hard_limit"] = 2 * WATCHDOG["rss_soft_limit"]
|
| # Not set by default:
| WATCHDOG["load1_limit"] = <1-minute load average limit>
| WATCHDOG["load5_limit"] = <5-minute load average limit>
| WATCHDOG["load15_limit"] = <15-minute load average limit>

The "rss_soft_limit" and "rss_hard_limit" defines the memory limits, in bytes.
The load average limit settings defines the limit at which to start warning
about high system load.  Any one, or all three, can be set to different values.
Note: the configured value is multiplied by the number of CPUs in the system
before being compared to the actual system load.

Extensions
----------

Critic supports an extension mechanism that allows users of the system to extend
Critic's functionality in various ways.  Extensions are written in ECMAScript
(executed in a custom interpreter based on the V8 ECMAScript engine) and have
access to a high-level API exposing most information in Critic's database.
Extensions can for instance be used to integrate Critic and other systems, such
as build servers or issue tracking systems.

The extension mechanism is not enabled in a newly installed Critic system.  The
system administrator needs to use the <code>extend.py</code> script to prepare
and enable the extension mechanism.  This script installs additional required
software, downloads (from GitHub) the source code used to build the ECMAScript
interpreter, builds and installs it, and finally modifies Critic's configuration
to enable the extension mechanism.

To enable the extension mechanism, simply run this command as root:

| python extend.py

For general information about the extension mechanism and how to create
extensions, see the
  <a href="/tutorial?item=extensions">extensions tutorial</a>
and the
  <a href="/tutorial?item=extensions-api">extensions API tutorial</a>.

Security considerations
-----------------------
Extensions are scripts running as the Critic system user, and effectively has
access to all information in Critic's database and Git repositories, and to all
files in the system's file systems that the Critic system user has access to.
In addition, they can start child processes, and thus do essentially anything
the Critic system user can do.  Untrusted users should obviously not be allowed
to add extensions to a Critic system.

To add extensions to a Critic system, a user needs shell access to the system,
or at least some way to create directories and files in their $HOME directory.
Typically, an untrusted user should not be given shell access to the system
anyway, so not allowing them to add extensions is somewhat a secondary concern.

  Hint: As mentioned above, users can be given access to Git repositories over
  SSH without giving them shell access, by setting the system users' shell to
  <code>git-shell</code>.

If users must be given shell access to the system, but not be allowed to add
extensions to it, it is possible to disable "user extensions" -- extensions
added by regular users -- by setting the configuration option
<code>USER_EXTENSIONS_DIR</code> in
  <code>%(configuration.paths.CONFIG_DIR)s/configuration/extensions.py</code>
to the value <code>None</code>.  With that configuration, only users with write
access to the directory
  <code>%(configuration.paths.DATA_DIR)s/extensions/</code>
can add extensions to the system.

Review branch archival
----------------------

For a description of this mechanism, see the [tutorial on the
subject][archival-tutorial].

A system administrator can tweak the mechanism by changing the system-wide or
per-repository defaults for the settings
CONFIG(review.branchArchiveDelay.closed) and
CONFIG(review.branchArchiveDelay.dropped).  Individual users can override the
defaults, but only to achieve a longer delay, or disable archival entirely, for
reviews they own.

If review branch archival is not desired at all, it can also be disabled
system-wide by the system administrator by setting
<code>ARCHIVE_REVIEW_BRANCHES</code> in
<code>%(configuration.paths.CONFIG_DIR)s/configuration/base.py</code> to
<code>False</code>.

[archival-tutorial]: /tutorial?item=archival
