SDxmlrpc
========

Introduction
------------

SDxmlrpc is an `XML-RPC <http://en.wikipedia.org/wiki/XML-RPC>`_ server for
StructuredData. Most of the functions and commands SDpyshell offers, can also
be used with SDxmlrpc.

What is XML-RPC ?
+++++++++++++++++

This is a way to perform 
`remote procedure calls <http://en.wikipedia.org/wiki/Remote_procedure_call>`_.
When you call a remote procedure, your system transfers the name of the
procedure and it's parameters to program on your local host or a remote server.
The server then executes the procedure and sends the results back to your
system. In order to make these calls easy to use the implementation of remote
procedure calls makes them look very much like ordinary (local) procedure
calls. 

In XML-RPC the parameters and results of a procedure are encoded in strings by
using `XML <http://en.wikipedia.org/wiki/Xml>`_. XML-RPC also uses the well
known `HTTP (Hypertext Transfer Protocol) <http://en.wikipedia.org/wiki/Http>`_
for communication between client and server.

There are XML-RPC libraries for many programming languages, including Python,
Perl, Ruby, C, C++ and Java.

What is the difference between SDxmlrpc and SDpyshell ?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

SDpyshell can be used as an interactive shell and as an interpreter for
scripts.  It is intended to be used by humans. SDxmlrpc on the other hand is
intended to be used by *programs*. These programs may run on the same host or
on a different host than the SDxmlrpc server and can use it's functions much
like a library.

Basic function
--------------

SDxmlrpc implements most of the functions of SDpyshell for XML-RPC. Since there
are only small differences to the functions of SDpyshell, this documentation
often refers to the documentation there and mainly emphasizes the differences
to SDpyshell.

The StructuredDataContainer key
+++++++++++++++++++++++++++++++

Almost all commands and functions of the SDxmlrpc server work with a
:ref:`StructuredDataContainer <reference-StructuredDataContainer>` that is
provided as parameter. In SDpyshell, if this parameter is omitted, the commands
use a global variable that is a StructuredDataContainer. In SDxmlrpc, the
StructuredDataContainer must *always* be specified as parameter, it must not be
omitted. As a further difference to SDpyshell, the StructuredDataContainer is
only *referenced* by a *key* which is a string. This is needed since XML-RPC is
independent of a programming language, so we cannot transfer python references.

In order to isolate applications that use the SDxmlrpc server at the same time,
the *keys* to the StructuredDataContainer are generated by a random process. A
*key* can only be generated by calling function newsdc. It creates a new
StructuredDataContainer and returns it's *key* which is a string.

In order to preserve memory on the SDxmlrpc server, a StructuredDataContainer
gets deleted if it was not read or written to for a certain time.

Return values
+++++++++++++

The SDxmlrpc server only gives access to functions of the functional and the
text layer (see also :ref:`Command layers <SDpyshell-command-layers>`). Functions
of the functional layer return data structures where functions of the text
layer always return a string which may be a multi-line string.

Functions
---------

See :ref:`Commands and Functions <SDpyshell-commands>` for a detailed
description. Note that only commands whose names start with "fun." and "txt."
are accessible via XML RPC.

Examples
--------

Here are some very simple examples on how to use XML-RPC with various
programming languages. All examples require a running StructuredData XML-RPC
server. In order to start the XML-RPC server go to the directory "samples" then
issue this command::

  SDxmlrpc --precmdfile Xprecmd.txt --port 8000

The file Xprecmd.txt loads the file idcp_db.cache.SDCyml under the name
"idcp_db". It has this content::

  # create a named StructuredDataContainer:
  fun.namedsdc("idcp_db")
  # read idcp_db StructuredData file:
  fun.read("idcp_db.cache.SDCyml","","idcp_db")

The XML-RPC clients try to connect on port 8000 to the hostname of your host.
Note that this is *not identical* to "localhost". All examples determine the
name of your system before trying to connect. In all the examples, a query is
performed which is equivalent to the following SDpyshell command::

  find("id-data.*.names.devicename")

The output of the programs is always this::

  id-data.U125/1.names.devicename : U125IV
  id-data.U125/2.names.devicename : U125ID2R
  id-data.U139.names.devicename   : U139ID6R
  id-data.U2.names.devicename     : U2IV
  id-data.U3.names.devicename     : U3IV
  id-data.U4.names.devicename     : U4IV
  id-data.U41.names.devicename    : U41IT6R
  id-data.U48.names.devicename    : U48IV
  id-data.U49/1.names.devicename  : U49ID4R
  id-data.U49/2.names.devicename  : U49ID8R
  id-data.UE112.names.devicename  : UE112ID7R
  id-data.UE46.names.devicename   : UE46IT5R
  id-data.UE49.names.devicename   : UE49IT4R
  id-data.UE52.names.devicename   : UE52ID5R
  id-data.UE56/1.names.devicename : UE56ID6R
  id-data.UE56/2.names.devicename : UE56ID3R
  id-data.UE56R.names.devicename  : UE56IV
  id-data.Ubonsai.names.devicename: U1IV

Python
++++++

This is the example query in python (file "xmlrpc-python.py" in directory
"samples")::

  #!/usr/bin/env python
  
  # How to start the server:
  # cd samples
  # SDxmlrpc --precmdfile Xprecmd.txt --port 8000
  
  import socket
  import xmlrpclib
  
  host= socket.gethostname()
  
  s = xmlrpclib.ServerProxy('http://%s:8000' % host)
  
  result= s.txt.find("id-data.*.names.devicename","","","idcp_db")
  
  print result

Perl
++++

This is the example in perl (file "xmlrpc-perl.py" in directory "samples"). You
need to have XML::RPC installed, it can be found here
http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm::

  #!/usr/bin/perl
  
  # This program needs XML::RPC:
  # http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm
  
  # How to start the server:
  # cd samples
  # SDxmlrpc --precmdfile Xprecmd.txt --port 8000
  
  use strict;
  
  use Sys::Hostname;
  use XML::RPC; 
  
  my $host= hostname;
  
  my $xmlrpc = XML::RPC->new("http://$host:8000/RPC2"); 
  
  my $result= $xmlrpc->call("txt.find","id-data.*.names.devicename","","","idcp_db"); 
  
  print "$result\n";

C
+++

This is the example in C (file "xmlrpc-c.c" in directory "samples"). You need
to have XML-RPC installed, which can be downloaded here
http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the
binary from the c-source with this command::
  
  gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c

Here is the program::

  /*
   
  This program needs XML-RPC:
  http://xmlrpc-c.sourceforge.net
  
  How to start the server:
  cd samples
  SDxmlrpc --precmdfile Xprecmd.txt --port 8000
  
  compile with this command:
  gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c
  */
  
  #include <stdlib.h>
  #include <string.h>
  #include <stdio.h>
  
  #include <unistd.h>  /* only needed for my_hostname */
  #include <netdb.h>   /* only needed for my_hostname */
  
  #include <xmlrpc-c/base.h>
  #include <xmlrpc-c/client.h>
  
  #define NAME "Xmlrpc-c Test Client"
  #define VERSION "1.0"
  
  #define PORT 8000
  
  static char *_get_hostname(void)
    {
      struct hostent* h;
      char hostname[1024];
      hostname[1023] = '\0';
      gethostname(hostname, 1023);
      h = gethostbyname(hostname);
      return strdup(h->h_name);
    }
  
  static void dieIfFaultOccurred (xmlrpc_env * const envP) 
    {
      if (envP->fault_occurred) 
        {
          fprintf(stderr, "ERROR: %s (%d)\n",
              envP->fault_string, envP->fault_code);
          exit(1);
        }
    }
  
  void print_unpack_string(xmlrpc_env *envP, xmlrpc_value *resultP)
    /* prints the string
     * DECREASES the reference counter of resultP 
     */
    {
      const char *ptr;
      xmlrpc_read_string(envP, resultP, &ptr);
      dieIfFaultOccurred(envP);
      puts(ptr);
      xmlrpc_DECREF(resultP);
    }
  
  int main(int const argc, const char ** const argv) 
    {
      char serverUrl[256];
      xmlrpc_env env;
      xmlrpc_value * resultP;
  
      sprintf(serverUrl, "http://%s:%d/RPC2", _get_hostname(), PORT);
      xmlrpc_env_init(&env);
      xmlrpc_client_init2(&env, XMLRPC_CLIENT_NO_FLAGS, NAME, VERSION, NULL, 0);
      dieIfFaultOccurred(&env);
  
      resultP= xmlrpc_client_call(&env, serverUrl, "txt.find", "(ssss)", 
                                  "id-data.*.names.devicename","","","idcp_db");
      dieIfFaultOccurred(&env);
      print_unpack_string(&env, resultP);
  
      xmlrpc_env_clean(&env);
      xmlrpc_client_cleanup();
      return 0;
  }

C++
+++

This is the example in C++ (file "xmlrpc-cpp.cpp" in directory "samples"). You need
to have XML-RPC installed, which can be downloaded here
http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the
binary from the c-source with this command::
  
  g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp
  
Here is the program::

  /*
   
  This program needs XML-RPC:
  http://xmlrpc-c.sourceforge.net
  
  How to start the server:
  cd samples
  SDxmlrpc --precmdfile Xprecmd.txt --port 8000
  
  compile with this command:
  g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp
  */
  
  #include <string.h>
  
  #include <unistd.h>  /* only needed for my_hostname */
  #include <netdb.h>   /* only needed for my_hostname */
  
  #include <iostream>
  #include <string>
  
  #include <xmlrpc-c/base.hpp>
  #include <xmlrpc-c/client_simple.hpp>
  
  #define PORT 8000
  
  static char *_get_hostname(void)
    {
      struct hostent* h;
      char hostname[1024];
      hostname[1023] = '\0';
      gethostname(hostname, 1023);
      h = gethostbyname(hostname);
      return strdup(h->h_name);
    }
  
  static std::string url(std::string host, int port)
    {
      std::string st= std::string("http://");
  
      st.append(host);
      st.append(":");
      st.append(std::to_string(port));
      st.append("/RPC2");
      return st;
    }
  
  int main(int const argc, const char ** const argv) 
    {
  
      std::string const serverUrl= url(std::string(_get_hostname()), PORT);
      xmlrpc_c::clientSimple myClient;
      xmlrpc_c::value *result_p;
  
      result_p= new xmlrpc_c::value();
      myClient.call(serverUrl, "txt.find", "ssss", result_p,
                    "id-data.*.names.devicename", "", "", "idcp_db");
  
      std::cout << std::string(xmlrpc_c::value_string(*result_p)) << "\n";
      delete result_p;
  
      return 0;
  }


Invoking SDxmlrpc
-----------------

Here is a short overview on the SDxmlrpc command line options:

--version             show program's version number and exit
-h, --help            show this help message and exit
--summary             print a summary of the function of the program
--info                Show ip, port and process ID on stderr.
--localhost           start server on 'localhost' instead of DNSDOMAINNAME.
                      In this case the server can only be contacted from
                      applications running on the same host.
--host                start server on HOST instead of DNSDOMAINNAME. This may 
                      be needed for hosts with more than one network interface.
--port=PORT           start xmlserver on port PORT
-p COMMANDS, --precmd=COMMANDS
                      specify COMMANDS to perform before any other action
--precmdfile=FILE     specify a FILE to execute before any other action
-M, --module          specify a MODULE to import at make its functions 
                      accessible by XMLRPC
-I, --searchpath      specify a DIRECTORY to prepend it to the module 
                      search path.
--no-locking          do not lock file accesses
--pidfile=PIDFILE     specify the PIDFILE where PID's of sub processes will
                      be stored
--kill                just kill old servers, do not start new ones.
--restart             restart the already running server

Precommands
+++++++++++

Precommands are commands that are executed at the start of the SDxmlrpc server
before any other command. These commands can be given as a command line
parameter (--precmd) or they can be read from a file (--precmdfile). A typical
application is to put the command to read a StructuredData file in a file and
provide it's name with --precmdfile. 

Extensions
++++++++++

These are user supplied python modules that can be loaded by the SDxmlrpc. The
module name (the filename without ".py") is provided with the command line
option "-M". In this case the python module is loaded and it's functions are
accessible with the module name as a prefix. 

You can use command line option "-I" in order to extend the search path for
extensions which are basically python modules. Keep in mind that extensions are
also searched in all paths specified by the "PYTHONPATH" environment variable.

Here is an example:

We have a file "myXext.py" with this content::

  import StructuredData.SDshelllibTxt as txt
  import StructuredData.SDshelllibFun as fun
  
  def ids():
      p= fun.paths("id-data.*",sdc="idcp_db")
      return fun.poppath(p, no=-1)
  
  def show_ids(formatspec="yaml"):
            return txt.format(ids(), formatspec)

We also have a file "Xprecmd.txt" with this content::

  # create a named StructuredDataContainer:
  fun.namedsdc("idcp_db")
  # read idcp_db StructuredData file:
  fun.read("idcp_db.cache.SDCyml","","idcp_db")
  

Now we start SDxmlrpc with "-M" to load the extension and with "--precmdfile" to load the 
sample StructuredData file from the "samples" directory::

  SDxmlrpc -M myext --precmdfile Xprecmd.txt --localhost --port 8000

To test the server we use the interactive python shell. We start python by
entering "python" on the command line::

  $ python
  Python 2.7.3 (default, Jul 24 2012, 10:05:39) 
  [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import xmlrpclib
  >>> import pprint
  >>> s = xmlrpclib.ServerProxy('http://localhost:8000')
  >>> pprint.pprint(s.myXext.ids())
  ['U125/1',
   'U125/2',
   'U139',
   'U2',
   'U3',
   'U4',
   'U41',
   'U48',
   'U49/1',
   'U49/2',
   'UE112',
   'UE46',
   'UE49',
   'UE52',
   'UE56/1',
   'UE56/2',
   'UE56R',
   'Ubonsai']
  
  >>> print s.myXext.show_ids("yaml")
  - U125/1
  - U125/2
  - U139
  - U2
  - U3
  - U4
  - U41
  - U48
  - U49/1
  - U49/2
  - UE112
  - UE46
  - UE49
  - UE52
  - UE56/1
  - UE56/2
  - UE56R
  - Ubonsai

Process management
++++++++++++++++++

When SDxmlrpc is started it is useful to know it's process id (PID) to be able
to restart the server by killing the old one and starting a new one. This is
done with the options --pidfile and --kill. --pidfile is used to specify the
name of a PID file, this file contains a line with the process id (PID) of the
server and the command that was used to start the server. When --pidfile is
provided, the process named in this file (and it's children) are killed first.
When SDxmlrpc is started, it's PID and command line are put to the PID file. If
you dont't want to restart an SDxmlrpc server but just want to kill the old
one, use --pidfile together with --kill.

Restarting the server
+++++++++++++++++++++

The server can forced do a complete restart. In this case it reloads the
precommand file and recreates all internal variables. This may be useful when
the StructuredData file on disk was changed and the server needs to reload the
file.

The server can either be restared by sending it signal SIGUSR1 like in::

  kill SIGUSR1 PID

where PID is the process ID, an integer, of the SDxmlrpc server, or by invoking::

  SDxmlrpc --pidfile PIDFILE --restart

when PIDFILE is the process ID file that was created when the server was started.
