Spring Batch - Reference
Spring Batch - Reference
The user can inspect the jobs that are known to the system. Jobs are either launchable or nonlaunchable (in the screenshot they are all launchable). The distinction is that a launchable job is defined and configured in the application itself, whereas a non-launchable job is detected as state left by the execution of a job in another process. (Spring Batch uses a relational database to track the state of jobs and steps, so historic executions can be queried to show the non-launchable jobs.)
Launchable jobs can be launched from the user interface with job parameters provided as name value pairs, or by an incrementer configured into the application.
Once a job is executing, or has executed, this view can be used to see the most recent executions, and a brief summary of their status (STARTED, COMPLETED, FAILED, etc.). Each individual execution has a more detailed view (shown above), and from there the user can click down to a view of each of the step executions in the job (only one in this case). A common reason for wanting to do this is to see the cause of a failure.
The top of the step execution detail view shows the history of the execution of this step across all job executions. This is useful for getting a statistical feel for performance characteristics. A developer running a job in an integration test environment might use the statistics here to compare different parameterisations of a job, to see what effect is of changing (for instance) the commit interval in an item processing step. The bottom of the step execution view has the detailed meta-data for the step (status, read count, write count, commit count, etc.) as well as an extract of the stack trace from any exception that caused a failure of the step (as in the example shown above).
A job that is executing can be stopped by the user (whether or not it is launchable). The stop signal is sent via the database and once detected by Spring Batch in whatever process is running the job, the job is stopped (status moves from STOPPING to STOPPED) and no further processing takes place.
2. Features
2.1. Installing Spring Batch Admin
There is a sample project in the distribution - it's a regular war file project which you can build with Maven ( mvn install and look in the target directory). The sample can also be created with a couple of clicks in the SpringSource Tool Suite (STS): go to File->New->Spring Template Project , and select the Spring Batch Admin Webapp from the list (if it isn't there upgrade STS). In STS you can right click on the project and Run As->Run On Server choose Tomcat or tc Server, and the app will deploy and open a web browser (e.g. localhost:8080/spring-batch-admin-sample). Check out the features for launching jobs and inspecting job executions. If the STS project looks out of date (look at the version numbers in the pom.xml) then you might be able to upgrade by changing the POM, or else you may have to install a nightly update of STS or wait for the next release.
This page is a list of URLs, or URL patterns, along with the HTTP method (GET, POST, etc.) that you need to use to carry out an operation on that resource. If the URL is a pattern it has braces ({}) in it to represent the value of a parameter that needs to be supplied by the user. Users of Spring REST support in MVC will recognise this convention, and also the idea that a GET represents a request for information, and a POST is an "insert" operation. So, for example, to request a list of all running Job executions you must GET the resource at /jobs/executions (relative to the batch/ servlet in the application). And to launch a new job execution you POST to /jobs/ jobName (with "{jobName}" replaced with the name of a launchable job). Links on the home page that are not parameterised and require a GET method are clickable in the browser and take you to that page in the application. The others are all navigable from there.
2.3. Menus
The basic menus for the application out of the box are - Home - the URL list or site map listing the interface for the application - Jobs - list and launch jobs. Only jobs which are either configured in the application or have been executed in other processes against the same database show up here. - Executions - view and interact with running Job and Step executions. - Files - upload and list input files and configuration files for jobs. You can learn how to extend the content of the application including the menus in a later section of this guide.
from the UI application and add a .json suffix. All the JSON endpoints are listed on the home page of the sample application. The examples below all use curl from the command line, but the same resources could be used with an Ajax request from a browser, or with Spring's RestTemplate (see the integration tests in the project source code for examples of the latter).
$ curl -s http://localhost:8080/spring-batch-admin-sample/batch/jobs.json {"jobs" : { "resource" : "http://localhost:8080/spring-batch-admin-sample/batch/jobs.json", "registrations" : { "infinite" : { "name" : "infinite", "resource" : "http://localhost:8080/spring-batch-admin-sample/batch/jobs/infinite.json", "description" : "No description", "executionCount" : 0, "launchable" : true, "incrementable" : false }, "job1" : { "name" : "job1", ... }, "job2" : { "name" : "job2", ... } } } }
As you can see from the example above, the information listed for JSON clients is the same as for HTML clients. A JSON response always contains a link to the resource that it came from, and also links to other interesting resources, e.g.
"resource" : "http://localhost:8080/spring-batch-admin-sample/batch/jobs/job1.json", "name" : "job1", "jobInstances" : { "0" : { "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/job1/0/executions.json", "executionCount" : 1, "lastJobExecution" : "/spring-batch-admin-sample/batch/jobs/executions/0.json", "lastJobExecutionStatus" : "FAILED", "jobParameters" : { "run.count(long)" : "0" } } } } }
The result here is a list of all the job instances for the job with name "job1". Each instance is identified as a JSON key whose value is the job instance id ("0" in the example above), and contains a link to another resource that you can visit (GET) to drill down into the job executions.
$ curl -d jobParameters=fail=false http://localhost:8080/spring-batch-adminsample/batch/jobs/job1.json {"jobExecution" : { "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/executions/2.json", "id" : "2", "status" : "STARTING", "startTime" : "", "duration" : "", "exitCode" : "UNKNOWN", "exitDescription" : "", "jobInstance" : { "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/job1/1.json" }, "stepExecutions" : { } } }
The input is a set of JobParameters specified as a request parameter called "jobParameters". The format of the job parameters is identical to that in the UI (a comma or new-line separated list of name=value pairs). The output is a JSON object representing the JobExecution that is running. In the example, there are no step executions yet because they haven't started running. The resource link on line 2 can be used to inspect the result and poll for updates to the status and the step executions. E.g. (with another GET):
$ curl -s http://localhost:8080/spring-batch-admin-sample/batch/jobs/executions/2.json {"jobExecution" : { "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/executions/2.json", "id" : "3", "status" : "COMPLETED", "startTime" : "17:44:52", "duration" : "00:00:00",
"exitCode" : "COMPLETED", "exitDescription" : "", "jobInstance" : { "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/job1/2.json" }, "stepExecutions" : { "step1" : { "status" : "COMPLETED", "exitCode" : "COMPLETED", "id" : "3", "resource" : "http://localhost:8080/spring-batch-adminsample/batch/jobs/executions/2/steps/2.json", "readCount" : "5", "writeCount" : "5", "commitCount" : "6", "rollbackCount" : "0", "duration" : "00:00:00" } } } }
We start with a look at the Jobs view in the application. It shows a list of jobs that are either launchable or monitorable by the web application. Now the plan is to upload a new Job configuration and see this view change. So we start with the "Files" menu in the top navigation, and then click on "Configuration". This presents a simple form to upload a file. If we point to a Spring XML configuration file, the view looks like this:
We press the "Upload" button and the configuration is uploaded and parsed and the Job instances in the uploaded file are registered for launching: You can see a new entry in the job registry ("test-job") which is launchable in-process because the application has a reference to the Job . (Jobs which are not launchable were executed out of process, but used the same database for its JobRepository , so they show up with their executions in the UI.)
A common variant of the basic upload use case is to upload a modification of an existing Job configuration, instead of a brand new one. This is common because one often needs to tweak the runtime parameters, especially when performance testing. For instance we could change the a href="http://static.springsource.org/springbatch/trunk/reference/html/configureStep.html#commitInterval"commit-interval /a in one of the Steps in a Job so that more (or less) items are grouped in a chunk and committed together during a repetitive item-processing Step . This might affect performance and the Job might run faster overall because of more efficient use of transactional resources. We would be able to measure and verify this directly by launching the Job from the application and inspecting the result (because the web interface is resource oriented, this could also be automated using a simple shell script based on curl ).
<bean id="batchMBeanExporter" class="org.springframework.batch.admin.jmx.BatchMBeanExporter"> <property name="server" ref="mbeanServer" /> <property name="jobService"> <bean class="org.springframework.aop.framework.ProxyFactoryBean"> <property name="target" ref="jobService" /> <property name="interceptorNames" value="cacheInterceptor" /> </bean> </property> <property name="defaultDomain" value="spring.application" /> </bean>
If you start the sample application with the usual command line options to expose the MBeanServer, e.g.
then launch one of the jobs, and look at the MBeans in JConsole or your favourite JMX client, you will see the BatchMBeanExporter . As soon as you inspect its attributes, you will also see Job and Step Executions as well (they are lazily registered only when the basic attributes are first inspected): The BatchMBeanExporter needs a reference to the JobService because it uses the JobRepository to extract most of its metrics. It also has a basic cache for the metric data, to prevent it having to go back to the repository every time it is polled. The default cache settings have an expiry timeout of 60 seconds (so new data will show up in JMX every 60 seconds). The cache can be reconfigured by overriding the bean definition for the cacheInterceptor bean.
org.springframework.batch:type=JobExectution,name=job1
where job1 is the job name. Metrics include information about the latest execution (status, duration, exit description, etc.) and also some basic summary information (number of executions, number of failures etc.).
spring.application:type=JobExectution,name=job1,step=step1
where job1 is the parent job name, and step1>> is the step name. Because of the form of the <<<ObjectName , most JMX clients present step execution metrics as a child of the job they belong to, making them easy to locate. Metrics include information about the latest execution (status, duration, commit count, read and write counts, etc.) and also some basic summary information (number of executions, number of failures etc.).
<bean id="j1.step1.monitor" class="org.springframework.batch.admin.jmx.StepExecutionServiceLevelMonitor"> <property name="jobName" value="job1"/> <property name="stepName" value="step1"/> <property name="upperThreshold" value="30000"/> <property name="defaultDomain" value="spring.application"/> </bean>
spring.application:type=GaugeMonitor,name=job1.step1.monitor
which can be exposed through the MBeanServer using the standard Spring context shortcut:
When a step is taking too long to execute, a JMX notification is sent automatically by the MBeanServer to any clients that have registered for notifications. The notification comes with an embedded event in the form:
where job1.step1.monitor is the name of the Spring bean definition used to define the monitor.
3. Customization
3.1. Environment Settings
The most likely thing you will want to customize is the location of the database. Spring Batch Admin ships with an embedded HSQLDB database, which is initialized on start up. - To change the database type add a file to the application classpath called batch[type].properties , where [type]is the database type you want to use (e.g. mysql , oracle , db2 ). Copy the contents of the batch-hsql.properties from the Manager jar and change the values to suit your environment. Then launch the application with a system property DENVIRONMENT=[type]. - To stop the database from being wiped and re-created on start up just set batch.data.source.init=false (in the properties file or as a System property). N.B. The use of the environment variable to switch on the database type is only a suggestion. You can change that and other things by overriding and adding configuration fragments to the Spring application context. See below for details.
- Bean ID : dataSource - Default : DBCP BasicDataSource with placeholders for common properties. - Override Scenarios : Change to another implememtation, or add additional placeholders.
<bean id="jobLauncherTaskExecutor" class="org.springframework.core.task.SyncTaskExecutor"/> - Change the pool parameters. Note that these settings can be changed at runtime using JMX, if the task executor is exposed explictly.
- Bean ID : throttledTaskExecutor - Default : a task executor with pool-size=600 and rejection policy CALLER_RUNS and infinite throttle limit. - Description : Not used anywhere in the manager, but provided as a convenience for applications that provide their own jobs and steps with concurrent behaviour. - Override Scenarios : The sample contains one job with a concurrent step ("job2"), and it works by creating a local taskExecutor bean inheriting from the root definition, and adding a throttle limit:
<bean id="taskExecutor" parent="throttledTaskExecutor"> <property name="throttleLimit" value="100"/> </bean> This task executor is then injected into a step in the sample which has the effect of limiting the concurrency of the step to 100, but within overall limits set globally by the bootstrap context, and without discarding or failing any tasks that are submitted. The throttle limit is set to 100 so that if this job is the only one running it will hardly be affected by the throttle, but if it is contending with other jobs, then they will all eventually run in the thread pool provided by the bootstrap (with 600 threads by default). The throttled executor itself delegates to a pool which can be modified independently by overriding its bean definition poolTaskExecutor , e.g.
<task:executor id="poolTaskExecutor" pool-size="200" rejectionpolicy="CALLER_RUNS"/> changes the pool size to 200 (from 600), limiting all jobs that use it to at most 200 threads. Your mileage may vary with these thread pool settings depending on the workload that is executed. It is worth experimenting with the parameters to see if you can affect the overall throughput or execution times of your multi-threaded jobs.
- Bean ID : jobLoader - Default : a AutomaticJobRegistrar with path locations set to classpath*:/METAINF/spring/batch/jobs/*.xml . - Override Scenarios : - Changing the path locations maybe (trivial but not that useful). - Change the inheritance strategy for post processors. By default the AOP configuration and the PropertyPLaceHolderConfigurer instances from the root context are copied into the child context created from the Job configuration files. You can change this behaviour just by providing your own versions in addition in the child context, or to make global changes you can override the jobLoader with one that copies different post processors down into the child, or one that does something completely different. - Override it to a dummy bean for the purposes of an integration test, so that it doesn't try to create multiple instances of the Job that you want to test. The sample application has an example. In JobExecutionTests-context.xml we want to load the Jobs explicitly into the test context, and prevent them from being loaded again by the job loader, so we do this:
<!-- prevent loading of other jobs by overriding the loader in the main bootstrap context --> <bean id="jobLoader" class="java.lang.String"/>
and use the same conventions to override and extend as you would for a web application. If you do that then in particular for a standalone application that runs a single job in its own process, you will want to override the jobLauncherTaskExecutor as described above, and possibly also the jobLoader in addition to providing data source properties. Here's an example configuration file to drive a job:
<beans ...> <import resource="classpath*:/META-INF/spring/batch/bootstrap/manager/*.xml" /> <import resource="classpath*:/META-INF/spring/batch/jobs/*.xml" /> <!-- ensure job runs in foreground thread --> <bean id="jobLauncherTaskExecutor" class="org.springframework.core.task.SyncTaskExecutor"/> <!-- prevent loading of other jobs by overriding the loader in the main bootstrap context --> <bean id="jobLoader" class="java.lang.String"/> </beans>
With the job defined in /META-INF/spring/batch/jobs/*.xml you can use the CommandLineJobRunner from Spring Batch to load the file above, and launch the job by name, e.g.
<import resource="classpath*:/META-INF/spring/batch/bootstrap/**/*.xml"/> <import resource="classpath*:/META-INF/spring/batch/override/**/*.xml"/> Note that the "override" location has no files in it in Spring Batch Admin distribution. This is a placeholder for users to add their own content. - The UI child context is loaded in the same way from
<import resource="classpath*:/META-INF/spring/batch/servlet/**/*.xml"/> N.B. the child context by default only loads when you visit the first page in the application. You should be able to see which files it actually loads from the log statements that come out on the server console.
- The job contexts are loaded by a special component ( ClasspathXmlJobLoader from Spring Batch) in the parent context. It looks for individual files in the pattern classpath*/METAINF/spring/batch/jobs/*.xml and loads each file individually as its own self-contained context, registering any instance of Job it finds in the JobRegistry at the top level. The job contexts are loaded in this way so that you can provide your own jobs in multiple (possibly many) JAR files included in the application, but you do not have to worry too much about duplicate bean definitions because only the Job names have to be unique. A job context inherits the AOP and PropertyPlaceholderConfigurer settings from the root context (not the standard behaviour of a child context) as a convenience, but you can add your own settings to apply locally as well.
loads a resource bundle (as per java.util.Resource ) from classpath:messages.properties . See the Spring MVC documentation for more detail on how message sources work. The message source is a set of key-value pairs where the key is provided by the application and the value is what you want to be rendered. To find all the possible key values you would have to inspect all the Freemarker templates in the resources and manager jars, and also the view bean definitions (where for example the page titles are defined) and controller definitions (for error codes). (A complete list of message keys would be a great addition to the project. If anyone wants to automate the compilation of that list please feel free to make a contribution.)
<bean name="jobs" parent="standard"> <property name="attributes"> <props merge="true"> <prop key="body">/manager/jobs/jobs.ftl</prop> <prop key="titleCode">jobs.title</prop> <prop key="titleText">Spring Batch Admin: Jobs</prop> </props> </property> </bean>
The only mandatory property in this is the "body", which is the location of a Freemarker template for the main body (business content) of the page. Freemarker templates by default have paths relative to either classpath:/org/springframework/batch/admin/web or /WEB-INF/web (either works but the WEB-INF location is dynamically reloadable at development time). The other properties shown above are to override the page title: the "code" is a key in the Spring MessageSource and the "text" is the default value if the code cannot be resolved.
. To override the settings here you could put a modified copy of those files in your WAR under /styles at the top level of your applications (the Spring JS ResourceServlet is used to search the context and all jars in that order). Or you could modify the standard view bean or its template (see below) to import a style sheet from a different location.
<bean name="jobs" parent="standard"> <property name="attributes"> <props merge="true"> <prop key="body">/manager/jobs/html/jobs.ftl</prop> <prop key="titleCode">jobs.title</prop> <prop key="titleText">Spring Batch Admin: Jobs</prop> </props> </property> </bean> <bean name="jobs.json" parent="standard.json"> <property name="attributes"> <props merge="true"> <prop key="body">/manager/jobs/json/jobs.ftl</prop> </props> </property> </bean>
To override one or other of these you would need to create a Spring config file in METAINF/spring/batch/servlet/override/*.xml and copy the definitions above replacing any of the properties as desired.
In the JARs the templates are located in sub-directories of org/springframework/batch/admin/web , and you can override them by putting files with the same name and relative path in WEB-INF/web . For example, the whole look and feel would change if you added a file WEBINF/web/layouts/html/standard.ftl to your application. Files in the WEB-INF/web location are also reloadable at development time.