Patrick's Playground

Lately I’ve been playing around with the CloudVPS ObjectStore, which is currently in beta phase. This blogpost shows the options of this ObjectStore in a practical way and concludes with a summary of commands you can use yourself to interact with it and some ideas. For this post, I assume you are familiar with cURL, REST and HTTP headers.

OpenStack

 

First just a small note on OpenStack: OpenStack is an alternative for AWS, which is pretty large already, but still gaining more ground. The API is a RESTful and stateless. As a result of this, you have to send your access token with every request, which we will see later.

OpenStack itself is divided into a couple of components, which have codenames. The most recent version of OpenStack is divided into the following components:

• Keystone (Authentication and authorization)
• Nova (Computing)
• Swift (ObjectStore)
• Glance (ImageStore)
• Horizon (Dashboard)

(To prevent any confusion: Glance is an image store for server OS images, not for graphical images).

 

Outline

 

The large picture of the operations we will be performing can be found here:

https://www.cloudvps.com/files/kb/objectstore-communication-diagram-big.png

In this post, we execute step 1 and 2 one time only, but steps 3 and 4 repeatedly.

 

Authenticating

 

As mentioned before: the requests to the ObjectStore need an access token. Therefore, we first need to authenticate with the service. We are going to authenticate to Keystone by using our username and password to get ourselves an access token.

On the CloudVPS website, it is mentioned that the request should look like this:

curl -v -H "Content-type: application/json" -d '{"auth": {"passwordCredentials":{"username": "objectstore@cloudvps.com", "password": "s3cr3tp@ssw0rd"},"tenantName":"tenanthandle-objectstore"}} 'https://os.cloudvps.com/v2.0/tokens

 

However, we don’t know our TenantName. Fortunately we can leave it out, which results in the following request:

curl -v -H "Content-type: application/json" -d '{"auth": {"passwordCredentials":{"username": "objectstore@cloudvps.com", "password": "s3cr3tp@ssw0rd"}}}' https://os.cloudvps.com/v2.0/tokens

 

If you know HTTP headers, you can already predict that in case you fail to authenticate you will get back an HTTP 401, Not Authorized.

If all goes well we get back an HTTP 200, OK with all information we need for the calls we would like to make:

{"access": {"token": {"expires": "2012-08-14T14:53:13Z", "id": "513a51e8c0d749aaac5ec31e83b1a756", "tenant": {"enabled": true, "id": "tenantId01", "name": "tenantId01"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "https://og.cloudvps.com/", "region": "AMS-01", "internalURL": "https://og.cloudvps.com:443/v1/AUTH_tenantid01", "publicURL": "http://tenantid01.og.cloudvps.com:80"}], "endpoints_links": [], "type": "object-store", "name": "swift"}], "user": {"username": "Patrick van Kouteren", "roles_links": [], "id": "PATRICK01", "roles": [{"id": "swiftoperator", "name": "swiftoperator"}], "name": "Patrick van Kouteren"}}}

 

This information contains our access token (513a51e8c0d749aaac5ec31e83b1a756), its expiry date (14th of August, 14:53:13) and information about or object store: the endpoints. The internalURL is basically the root dir of the storage. The publicURL is the public root URL.

 

Container operations

 

The operations we are able to perform on a container are listed here: http://docs.openstack.org/api/openstack-object-storage/1.0/content/storage-container-services.html

Let’s start off by getting a list of the containers in our object store (which we expect to be none..) A list of the containers can be retrieved by sending a GET request to the root directory of your object store. Note that you have to add the access token to this request in a header called ‘X-Auth-Token’, like so:

curl -v -H "X-Auth-Token: 513a51e8c0d749aaac5ec31e83b1a756" https://og.cloudvps.com:443/v1/AUTH_tenantid01

 

The response we get looks like this:

< HTTP/1.1 204 No Content
< X-Account-Object-Count: 0
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 0
< Accept-Ranges: bytes
< X-Trans-Id: txba227e388211489496173b2315958189
< Content-Length: 0
< Date: Mon, 13 Aug 2012 15:05:10 GMT

 

The object count, bytes used and container count are all 0, which means that the object store completely empty.

Let’s make an end to this by adding a container to it. For this, we need the PUT header. When you don’t define a header with curl, it default to GET, so for this operation, we need to specify the PUT header explicitly by using the -X argument of curl. Let’s create a container ‘Foo’. Again, we need to pass the X-Auth-Token for authentication.

curl -v -X PUT -H "X-Auth-Token:513a51e8c0d749aaac5ec31e83b1a756" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

We get back:

< HTTP/1.1 201 Created
< Content-Length: 18
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: txe77616b2d9224b6495c13e4d1f4ce99f
< Date: Mon, 13 Aug 2012 15:12:19 GMT
<

201 Created

 

Woohoo, we just created a container. Let’s check it by again sending the list container command:

curl -v -H "X-Auth-Token:513a51e8c0d749aaac5ec31e83b1a756" https://og.cloudvps.com:443/v1/AUTH_tenantid01

 

The response this time is:

< HTTP/1.1 200 OK
< X-Account-Object-Count: 0
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 1
< Accept-Ranges: bytes
< Content-Length: 4
< Content-Type: text/plain; charset=utf-8
< X-Trans-Id: txab0642d73bba4a9f84762938963fb978
< Date: Mon, 13 Aug 2012 15:13:16 GMT
<

Foo

 

A nice thing about these containers and HTTP requests is that we can add metadata to them. This metadata is added by using the HTTP headers. Every piece of metadata in a header should be preceeded by ‘X-Container-Meta-‘. Let’s add some metadata to the container ‘Foo’. We do this by using the PUT command. Here we’re adding the metadata item ‘description’ with a value:

curl -v -X PUT -H "X-Auth-Token:c6959eaa059a414c8ad0c4d3a0b2ade8" -H "X-Container-Meta-Description:Contains my family photo album" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

If all goes well, we get back a 202 response, letting us know that the request is accepted for processing:

< HTTP/1.1 202 Accepted
< Content-Length: 58
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx446267a2ad8549ae8676803cdcef1b25
< Date: Wed, 15 Aug 2012 15:09:42 GMT
<

202 Accepted

The request is accepted for processing.

 

If we would send this request again, we would get exactly the same response. This is because this operation is idempotent: regardless of how many times a given method is invoked, the end result is the same.

Now for retrieving information about a container, like the metadata, we can use the HEAD header in a request on the container url, like so:

curl -v -X HEAD -H "X-Auth-Token: c6959eaa059a414c8ad0c4d3a0b2ade8" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

The response shows the metadata we set earlier:

< HTTP/1.1 204 No Content
< X-Container-Object-Count: 0
< X-Container-Bytes-Used: 0
< X-Container-Meta-Description: Contains my family photo album
< Accept-Ranges: bytes
< Content-Length: 0
< X-Trans-Id: tx46bae217a4254d7494543c6285b80571
< Date: Wed, 15 Aug 2012 15:14:54 GMT

 

Adding files

 

With cURL we can easily send files by using the -T handle. With the next command we can send a file to the ‘Foo’ container:

curl -v -X PUT -H "X-Auth-Token:c6959eaa059a414c8ad0c4d3a0b2ade8" -T /Users/patrick/Desktop/chuck.jpg https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo/chuck.jpg

 

Again, if all goes well, we get back a 202 header and the message that our request is accepted:

< HTTP/1.1 202 Accepted
< Content-Length: 58
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx70c1558b053e4d2f9b06621f3b234a39
< Date: Wed, 15 Aug 2012 15:27:13 GMT
<

202 Accepted

The request is accepted for processing.

 

Now, let’s check if the file is present in the container. We could to this by performing a HEAD on the container:

curl -v -X HEAD -H "X-Auth-Token: c6959eaa059a414c8ad0c4d3a0b2ade8" https://og.cloudvps.com:443/v1/AUTH_ppk000013/Foo

 

This will give us the info like so:

< HTTP/1.1 204 No Content
< X-Container-Object-Count: 1
< X-Container-Bytes-Used: 164898
< X-Container-Meta-Description: Contains a monkey photo album
< Accept-Ranges: bytes
< Content-Length: 0
< X-Trans-Id: tx6da53a8df85c426fbb80c7c56e3611c2
< Date: Tue, 21 Aug 2012 15:06:46 GMT

 

As we can see: object count is 1.

But perhaps we could’ve better done a GET request on the container. Not only will this return the same information, it will also return a list of files so we can actually verify the image name. Like so:

curl -v -H "X-Auth-Token: c6959eaa059a414c8ad0c4d3a0b2ade8" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

The result is almost the same, but with the file(s) in the container listed in the response:

< HTTP/1.1 200 OK
< X-Container-Object-Count: 1
< X-Container-Bytes-Used: 164898
< X-Container-Meta-Description: Contains a monkey photo album
< Accept-Ranges: bytes
< Content-Length: 10
< Content-Type: text/plain; charset=utf-8
< X-Trans-Id: tx1a116af90f3a42b5a01f0f12dd364988
< Date: Tue, 21 Aug 2012 15:07:31 GMT
<

chuck.jpg

 

Container rights: public access

 

By default, a container is private. If you’d try to point your browser to the publicURL which we received upon authentication, you’d receive a HTTP 401 “Unauthorized”, as well as when requesting the actual file. To be able to see the image by using a web browser (or retrieving it in any other way..), we need to allow reading. The access control (ACL) is a specific metadata header. Example 4.15 of http://docs.openstack.org/api/openstack-object-storage/1.0/content/special-metadata-acls.html describes how to give read access and listing to anyone.

Let’s first give just read access to the file. We do this by adding the X-Container-Read header. Note that we send this request to the container URL. We can only make a container public or private, not an object in that container! There’s no such thing as having a private container with some private files and some public. The request:

curl -v -X PUT -H "X-Auth-Token: c6959eaa059a414c8ad0c4d3a0b2ade8" -H "X-Container-Read: .r:*" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

As a response we again get a 202 header indicating that the request is being processed.

 

< HTTP/1.1 202 Accepted
< Content-Length: 58
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx2d7792d30283405ab11f7e8da04ea1f1
< Date: Mon, 27 Aug 2012 09:07:36 GMT
<

202 Accepted

The request is accepted for processing.

 

By now, the container contents are public. This means that you can point your browser to the public url (which in this example was http://tenantid01.og.cloudvps.com and append the path to the file to it (resulting in http://tenantid01.og.cloudvps.com/Foo/chuck.jpg) and the image will show up in your browser. (Of course you could also call this through cURL)

You could also allow public listing of the container. For this, you can use the ‘.rlistings’ value for the X-Container-Read header. When allowing listing the container publicly, everyone can point their browser to the container url and get a listing of all contents of the container in XML format.

Note that you cannot allow listing while now allowing read access to the container. This means we will need to allow both reading and listing the container. This sounds obvious, but for doing this you will need to send two ‘rights’ to the container. Applying only the right to do public listing will result in the request being accepted, but not working when trying to list a container in your browser.

So, we send the request for applying the new rights (Reading and listing):

curl -v -X PUT -H "X-Auth-Token: c3d77f5302dlca22bea12f384076fb10" -H "X-Container-Read: .r:*,.rlistings" https://og.cloudvps.com:443/v1/AUTH_tenantid01/Foo

 

(For the record: The result again is a 202, accepted)

When pointing your browser to the public url to the container (in this example http://tenantid01.og.cloudvps.com/Foo) you will receive a listing of the directory in XML format:

<container name="Foo">
<object>
<name>chuck.jpg</name>
<hash>033ac917e5c6cc0c4bddd725899703f9</hash>
<bytes>164898</bytes>
<content_type>image/jpeg</content_type>
<last_modified>2012-08-21T14:53:57.144230</last_modified>
</object>
</container>

 

Note that the meta data does not show up in your browser as it’s part of the headers!

 

Wrapping up

 

This concludes this small intro of working with the CloudVPS object store. I have not treated deletion operations here. If people are having difficulties performing these operations I can outline them in a next blogpost, but I think most of you will understand how to send a DELETE request after this post.

 

Final thoughts and next blogpost

 

Once you’re working with this Object store it feels natural, in the same way as you would work with a VPS on the command line. However, it’s a little more verbose and some more hassle is involved to actually get things to work. The URLs aren’t pretty as well. In a next post I will show how you can make your life a little easier regarding the URLs. Also I would like to compare the speed difference in terms of page loading time between serving content from the object store and serving it from the same location as the HTML / PHP file itself.

And, no promise on that yet, I’ll try to integrate the object store in a website deployment strategy.

Any comments, requests are highly appreciate. If you want to experiment for yourself, signup for the free beta by sending an e-mail to the sales department. You’ll get an object store of 200GB to play with for at least 3 months. Be sure to leave some feedback either via mail, on the blog article itself or tweet to @CloudVPS.