Skein comes with an informative Web UI. This displays information on:
What services are currently running in the application
Status of all current and past containers
Live links to logs for each service
Key-value pairs in the Key-Value Store
As well as links to user registered pages.
The Skein Web UI is usually accessed through the YARN Resource Manager Web UI.
The Overview page presents a summary view of the application and its contained services. Information contained here includes:
The application name, ID, and user
Links to the Application Master and Application Driver logs (if relevant)
Total memory and cores allocated for this application
Application progress (if set by the application)
A summary of the application services
Here we have two services -
icons in each row indicate for each service (in order):
How many containers are currently running
How many containers succeeded (exit code of 0)
How many containers were killed (stopped by the user)
How many containers have failed (non-zero exit code)
In this case,
my-first-service has 2 containers running, 1 container
succeeded, 1 container killed, and 1 container failed.
Opening the accordion for the service displays three tables, summarizing all pending, running, and completed containers.
If applicable, links to the live logs for each container are provided, as well as their current status and total runtime.
The Key/Value page contains information on all key-value pairs currently set in
the Key-Value Store. This can be useful for
debugging applications. Note that if a value is not UTF-8 encodable it is
Applications can register additional pages in the Web UI, which are available through the “Service Pages” dropdown. This can be useful for providing application specific dashboards.
Custom pages can be added with
WebUI.add_page(), removed with
WebUI.remove_page(), and queried with
Add a new proxied page to the Web UI.
Remove a proxied page from the Web UI.
Get all registered pages.
# Adds a page at <address_to_web_ui>/pages/custom
# The link in the dropdown will be "My Custom Page"
app.ui.add_page('custom', my_custom_page_address, link_name='My Custom Page')
# Get a map of all registered pages
# Remove the page with the route `custom`
Note that due to limitations in the YARN web application proxy,
GET requests can be successfully proxied through in most deployments
PUT requests was added in Hadoop 3). This means that modern
web components like websockets won’t work, nor will many RESTful web apps. Even
so, useful web pages can still be supported.
A few high level notes:
Since the YARN web application proxy doesn’t support websockets, we use an AjaxDataSource. This is less efficient than a
ColumnDataSource(which uses websockets), as it requires polling the server to get updates. For small amounts of data though this is sufficient and fine.
We choose a dynamic port, as we can’t know for sure what ports will be available on the deployed container.
The only Skein-specific bit is where the page is registered with the application:
# Register the page with the Skein Web UI. # This is the only Skein-specific bit app = skein.ApplicationClient.from_current() app.ui.add_page('price-dashboard', address, link_name='Price Dashboard')
The deployed page looks like:
YARN handles authentication but not authorization for application web pages. As such, users accessing the pages must successfully authenticate with YARN (either using kerberos or “simple” authentication, depending on cluster configuration). However, by default any successfully authenticated user will have access to the Skein Web UI.
If this is undesired, you’ll need to configure security using Access Control Lists (ACLs). The simplest version disables access to the Web UI for everyone but the application owner:
All requests to the Web UI are authenticated by YARN and authorized by Skein against the specified ACLs. This goes both for requests made from outside the cluster (through the YARN web ui) and inside the cluster with direct access to the Web UI (this is due to YARN’s provided security filter).
While the Skein Web UI is secured against both external and internal access by the YARN security filter, Custom Pages are currently only secured against access from outside the cluster (i.e. through the YARN Web UI). Users with SSH access to the cluster and knowledge of the internal address/ports used for custom web pages will currently have unrestricted access to these custom pages.
This is fixable, but deemed lower priority, as it’s only an issue in the case of malicious users that already have direct access to the cluster. If this is a problem for you, please let us know by filing an issue.