Tuesday, 4 June 2013

1,001 chained streaming-only replication instances

On my laptop I've managed to create 1,001 local chained (one-to-one) streaming-only (meaning no archive directory) asynchronous replication instances. The output of the status of the list is here: https://gist.github.com/darkixion/5694200

I also tested the promotion of the 1st standby to see if it would cope with propagation to the final 1,000th standby, and it worked flawlessly. This didn't work on my copy of Linux Mint without some adjustments to the kernel semaphores values, and it does take a while for all the standbys in the chain to reach full recovery. However, promotion propagation is very fast.

Try it for yourself (if you have enough RAM that is). You may find it quicker to use my pg_rep_test tool. Just don't do this manually... it'll take far too long.

Thanks to Heikki for putting in the changes that made this archiveless cascading replication possible. :)

Update: some figures

So looking at the logs, it's clear why it takes so long for all 1,000 standbys to come online; it tries to connect to its replication host every 5 seconds, so the delay between the host coming online and the standby coming online is up to 5 seconds. This potentially amounts to 5,000 seconds (about 83 mins) to ensure they're all online and receiving a streaming replication connection. A test of this shows it taking 46 minutes 25 seconds.

And as requested by Jonathan Katz (@jkatz05) I can tell you that the amount of time it takes for the promotion of the 1st standby to cause the 1,000th standby to switch to the new timeline (at least on my laptop with an SSD) is 1 minute 46 seconds, so a rate of 9.266 promoted instances per second. And as for actual data changes (in the case of my test, the creation of a table), it took about 6 seconds to reach the 1,000th standby. Re-tested with an insert of a row, and it's about the same again.

Monday, 27 May 2013

Accepted Google Summer of Code projects 2013

This year's accepted Google Summer of Code projects have been published. Among them are 3 accepted proposals for PostgreSQL which not only will be fine addition to PostgreSQL's set of features, but also helps introduce new students to our community for hopefully long-term contributions.

The projects that will be worked on are:

Cube extension improvement (Stas Kelvich)

Indexes on the cube data type (in the "cube" extension) tend to be on the large size, meaning maintenance of and accessing these indexes is expensive. The improvements this project aims to implement are in reducing the cost of indexes on cube data by using r-tree structures. In addition to this, PostgreSQL's relatively new K-Nearest Neighbour framework would serve to allow the creation of ordering operators for retrieving sorted data directly from the index, and ordering operators for kNN with different spatial norms.

UPDATE ... RETURNING OLD (Karol Trzcionka)

PostgreSQL can perform UPDATE statements and return the new row by using the RETURNING clause and referencing the columns you want. This project would introduce NEW and OLD aliases to provide the ability to reference not just the new row but also the old. This would allow for a before/after comparison of data.

Efficient KNN search through high-dimensional indexing with iDistance (Mike Schuh)

This will introduce a new indexing algorithm that utilises a high-dimensional space leading to more efficient K-nearest neighbour searches. Such indexes are an advantage over b-tree and r-tree which degrade with a modest increase in dimensions, whereas the iDistance algorithm has been demonstrated to remain well-performing and efficient.

Of course our students won't be left to work in isolation; they will also receive guidance from established community members specifically assigned to mentor them. We welcome Stas, Karol and Mike to the community, and hope not only that they are successful in their projects, but that they continue to contribute beyond this year's Google Summer of Code. Also thanks to Alexander Korotkov, David Fetter and Stephen Frost who will mentor this year's students. It's worth noting that Alexander was actually a GSoC student last year whose work on indexing on ranges made it into the upcoming 9.3 release.

Monday, 2 April 2012

Deprecated features in PostgreSQL - Past to present


If you have been using PostgreSQL for a long time, or you’re relatively new but have been following old instructions about how to use it, it’s possible that you’re using features that have been deprecated. The reason features disappear tend to be because they have been superseded by better features which cover the same functionality. It’s important to try to avoid using features which are destined to disappear if there’s a newer alternative. Also when planning an upgrade, it’s useful to know if a feature you’re using will suddenly break in the new version. Some of these features still continue to work, but have since been removed from documentation because they’re to be removed in a future release. Others have just been removed completely.

We’ll start off with the absurdly old and work our way to the present:

Version 6.2

timetravel contrib module

Does it still work?: No. This is ooooold and was last supported back in PostgreSQL 6.1.

What’s wrong with it?: This really dragged performance down and took up a huge amount of storage space. But the concept was pretty cool... being able to query data as it was at another time.

What to use instead: You can use triggers to implement a similar mechanism.

Version 6.4

char2/char4/char8/char16 data types

Does it still work?: No. These were removed way back in PostgreSQL 6.4. In fact I shouldn’t bother mentioning these, but you never know... someone *could* still be using them somewhere.

What’s wrong with it?: Not in the SQL standard and they’re no faster than using the ubiquitous char(n).

What to use instead: char(n)

Version 7.0

abstime data type

Does it still work?: This will still work, but it’s no longer documented as of PostgreSQL 7.0 and only intended to be used internally. Despite its name, it supports both date and time.

What’s wrong with it?: The range this data type provides is limited: ‘1901-12-14’ to ‘2038-01-19’. It also only has a resolution down to the second. Its behaviour is unfortunately like that of MySQL’s, in that if you insert an invalid value, it won’t fire an error. Instead you’ll just see ‘invalid’ as the value when you go to query it.

What to use instead: Since abstime supports timezone, the better alternative is using timestamp with time zone (timestamptz). It takes up more space (8 bytes instead of 4), but it has a far wider range: ‘4713 BC’ to ‘294276 AD’ and supports microsecond resolution.

reltime data type

Does it still work?: Yes, still works, but again, no longer documented as of PostgreSQL 7.0 and for internal use only.

What’s wrong with it?: This stores a date/time offset but only +/- 68 years. Again, this doesn’t error with values higher than this limit. Does it put ‘invalid’ in the column like abstime? No. Instead the value wraps around, so entering +70 years would result in a value of around -66 years. Not what you want. It also has a resolution down to the second.

What to use instead: The SQL standard equivalent of this kind of data type is interval, which PostgreSQL has. This does take up more space (12 bytes), but it’s range is absolutely huge: ‘-178000000 years’ to ‘+178000000 years’. This also has microsecond resolution. Interval can also handle relative time units; for example, adding a month to 15th February using interval will give you 15th March, but with abstime it has a fixed notion of a month being 30 days, so will give you 17th March (on a non-leap year). A year is also considered to be 360 days.

timespan data type

Does it still work?: No. This was deprecated back in PostgreSQL 7.0, and totally removed in PostgreSQL 7.3. If for any reason you’re using this, you’re *definitely* overdue an upgrade, and have been for many years.

What’s wrong with it?: It’s not in the SQL standard and was really just an alias for interval.

What to use instead: Just use interval.

psql/pg_dump’s -u option

Does it still work?: No, this was deprecated as far back as PostgreSQL 7.0 and removed in 8.3. You should definitely not be using this.

What’s wrong with it?: This option forced psql and pg_dump to prompt for the username and password before connecting to the database. Since prompting for a username is always optional, but prompting for a password may or may not be required (depending on authentication method), it didn’t make sense to glue both of these together.

What to use instead: It has been replaced by the -U option to specify the username, and the -W option to prompt for the password.

Version 7.1

getpgusername() function

Does it still work?: Yes, but it’s effectively deprecated as of PostgreSQL 7.1!

What’s wrong with it?: It’s no longer documented, and could be removed in a future release since it’s obsolete.

What to use instead: Call current_user instead, since getpgusername() is now just an alias for that.

Version 8.1

autovacuum contrib module

Does it still work?: No, as it was moved into core since PostgreSQL 8.1.

What’s wrong with it?: Nothing. Quite the opposite. It was considered so essential that it became part of the main codebase.

What to use instead: Nothing to worry about. Since it’s now in core, you get it out of the box without having to explicitly include it.

Version 8.2

mSQL-interface and tips contrib modules

Does it still work?: No, these were completely removed in PostgreSQL 8.2.

What’s wrong with it?: These were considered abandoned and unmaintained.

What to use instead: Nothing.

adddepend, dbase, dbmirror, fulltextindex, mac, ora2pg and userlock
contrib modules

Does it still work?: No, again, these were completely removed in PostgreSQL 8.2.

What’s wrong with it?: Most of these were moved to pgFoundry to be maintained separately.

What to use instead: These still exist on pgFoundry if you really want them (except for fulltextindex which has disappeared, and ora2pg which is on its own website), although they’re all now unmaintained (apart from ora2pg).

Version 8.3

automatic casting to text

Does it still work?: As of PostgreSQL 8.3, non-text data types are no longer implicitly cast to text. This is considered to be one of the major hurdles for some people migrating from earlier versions and the biggest cause of incompatibility.

What’s wrong with it?: Anyone who knows PostgreSQL well will know that it doesn’t like to throw any weirdness or odd behaviour your way. There are cases where implicit casting to text causes undesired results. For example: current_date < 2012-04-02 would result in both sides being automatically cast to text types, even though the date on the right-hand side would first be considered an integer (2012 minus 4 minus 2).

What to use instead: It’s always good practise to be explicit about data types when specifying literals. This will avoid any usual behaviour.

tsearch2 contrib module

Does it still work?: Yes, but it is deprecated as of PostgreSQL 8.3.

What’s wrong with it?: It has been superseded by changes in core with a few functional changes. It’s still kept around for backwards-compatibility.

What to use instead: Use the newer core functionality. There’s information on the tsearch2 contrib module page in the documentation on how to convert to the new functionality.

xml2 contrib module

Does it still work?: Yes, but it is deprecated as of PostgreSQL 8.3.

What’s wrong with it?: Nothing really, and it’s still around for backwards-compatibility, but there is newer XML functionality in core based on the SQL/XML standard.

What to use instead: Use the built-in XML features (xml data type, xml functions, xml parameters).

Version 8.4

pg_dump/pg_dumpall’s -d and -D options

Does it still work?: No, these were removed in PostgreSQL 8.4.

What’s wrong with it?: Such options were often mistaken for a database name parameter, but in fact it caused database dumps to output using insert statements rather than copy statements. This is significantly slower to restore, and cannot be adjusted after the fact.

What to use instead: If someone really did want to use these options intentionally, then the long name options of --inserts and --column-inserts are to be used instead.

Version 9.1

createlang/droplang client applications

Does it still work?: Only up until PostgreSQL 9.1.

What’s wrong with it?: Languages are now treated like extensions as of PostgreSQL 9.1.

What to use instead: Execute CREATE EXTENSION <language name> instead.

CREATE/DROP LANGUAGE

Does it still work?: Yes, but it’s no longer intended to be used by users, only extensions.

What’s wrong with it?: Languages are now considered to be extensions as of PostgreSQL 9.1.

What to use instead: You can install new languages by installing it as an extension with CREATE EXTENSION <language name>. And to remove it, use the DROP equivalent. If you’ve upgraded your cluster to 9.1 or above from a previous version, you will still have the language installed but not as an extension. You can, however, convert it to an extension by using: CREATE EXTENSION <language name> FROM unpackaged. You can then remove it later with DROP EXTENSION.

Version 9.2

=> operator

Does it still work?: Only up to PostgreSQL 9.1, but as of PostgreSQL 9.2, no, at least as far as hstore is concerned. This was actually deprecated in 9.0 and has emitted warnings about using it since then. This is most notably used in the hstore extension. You can still create this operator, but it will return a warning when you do so. Be prepared for this to be completely disallowed in a future release.

What’s wrong with it?: “=>” is reserved in the SQL standard for named function parameters, so it needs to be available for such functionality.

What to use instead: If you’ve been using this in hstore, then it will require changing text=>text to hstore(text,text). If you’ve been using it as a custom operator, you should change it to something else as at some point it will be prohibited.

Literal language name case-sensitivity

Does it still work?: If you're on 9.1 or below, yes, but as of 9.2 you won't get away with this anymore.

What’s wrong with it?: Language names should be treated like an identifier rather than a literal, and in general, literals are case-sensitive. There was special code that case-folded the language name so that the letter casing didn't matter, but this change is the first step in removing string literals as language names altogether.

What to use instead: Just don't use single quotes around language names when writing your functions at all, rather than just lower-casing them. Either use no quotes (as they're not needed for any core language) or use double-quotes as you would with any other identifier.

There are also lots of configuration parameters that have been removed, and here they are:

ParameterRemoved inWhy?
australian_timezones8.2Better generalised timezone configuration
preload_libraries8.2Renamed to shared_preload_libraries
bgwriter_all_percent8.3No longer necessary
bgwriter_all_maxpages8.3No longer necessary
bgwriter_lru_percent8.3No longer necessary
redirect_stderr8.3Renamed to logging_collector
stats_block_level8.3Now covered by track_counts
stats_command_string8.3Renamed to track_activities
stats_reset_on_server_start8.3pg_stat_reset() can be used instead
stats_row_level8.3Now covered by track_counts
stats_start_collector8.3Now always enabled
explain_pretty_print8.4No longer needed
max_fsm_pages8.4No longer needed as per-relation free space maps deal with this.
max_fsm_relations8.4No longer needed as per-relation free space maps deal with this.
add_missing_from9.0Always defaulted to ‘off’ so now permanently off.
regex_flavor9.0Always defaulted to ‘advanced’ so now permanently set to this.
custom_variable_classes9.2Considered better to remove it as it only causes more maintenance with minimal benefit.
silent_mode9.2Not necessary as can be achieved with pg_ctl -l or NOHUP.
wal_sender_delay9.2New latch infrastructure has now made this setting redundant.

Monday, 23 January 2012

Password restrictions

I've been changing a lot of my passwords lately so that none are alike. In fact they're so ridiculously random and lengthy that I wouldn't stand a chance remembering them. I've a means of obtaining these passwords though in a method known to me using encryption which requires 2 very different types of key, so remembering them isn't necessary.

However, after going around trying to change my passwords, I've noticed that many places impose restrictions upon how long a password may be, and what characters you're allowed to use.

For example, with PayPal, they absolutely require you to have at least 8 characters, but for some reason, it must not be any more than 20 characters. Why limit it at 20? Surely they hash the password anyway?... don't they? eBay have an identical restriction.

And then came my bank (a Dutch bank which shall remain nameless). They also said that it must be at least 8 characters, but no more than 12. It must also contain at least one lower-case letter, one upper-case letter, one number and one non-alphabetic character. But then it also may not use a speech mark ("), equals (=), tilde (~), less-than (<) or greater-than (>). So this not only suggests they're not hashing the password, but there's also probably some risk posted by those additional characters, meaning they may not be sanitising their data. Normally if that were the case I'd expect characters like a back-tick (`), semi-colon (;) or single-quote (') to be prohibited, but for some reason those are allowed in this case. *shrug*

Google seems to be better at this, but still imposes an arbitrary limit. In this case they require a minimum of 8 characters, and allow up to 100 characters. Why 100? Does it matter? Thankfully they allow any character you like, including spaces and all types of punctuation you can think of. But in addition to this, I have 2-factor authentication enabled, so I require a 6-digit pin generated by my phone, so the risk of someone logging into my account is minimised. Although I guess this is all moot when it comes to Google's application-specific passwords. This is necessary if you use 2-factor authentication when you wish applications to access your account (such as an Email client), because those applications can't ask you for the 6-digit pin. So Google generates a password specifically for that application which can be revoked whenever you like. Here's an example of what its generated passwords look like:

qihi jnmd irjb ytis

They always show up as just lower-case letters, and the spaces don't matter. So really, if you have application-specific passwords enabled, the security on your Google Account is only as strong as those passwords, and obviously the more you have the lower the security. Still, it's extremely unlikely anyone could get into your account using brute-force trying to find a code, but it's a shame Google doesn't allow you to enter an application-specific password of your choice.

I also recently set up an account with Good Old Games (gog.com). The limit there was 32 characters. I tried using special characters in the password but found that it caused the form to show the password field as invalid, but wouldn't say why. I figured out they only allow upper-case and lower-case letters and numbers.

Amazon fare better than the rest so far. Any characters you want, up to 128 characters.

The best I've seen is SpiderOak. I gave it a totally random mix of every type of character and a password 1024 characters long. It was fine with this. I don't know how many more characters it will take, but surely that should be enough.

So SpiderOak is the most secure out of this list. SpiderOak also can't recover your password because they don't store it. Apparently it's only used to decrypt your user data, which they can't access. In fact they boast a zero-knowledge policy, meaning they don't know your password or anything that you're storing, not even including file names.

On the opposite end is the Dutch bank who shall remain nameless. You'd expect a bank to have higher requirements than most, but it seems not. 8-12 characters? And some characters excluded? What are they playing at? They don't even offer 2-factor authentication. It's just username and password (no, not even account details).

So, anyone know why these companies place such restrictions on passwords? Does it present a denial-of-service avenue? Do they not hash it in case they want to provide the password to, say, the NSA? Any good reason?

Saturday, 7 January 2012

With a Little Help by Cory Doctorow

Look what arrived today (apologies for the poor quality pictures you're about to see).


Yes, something wrapped in a coffee burlap bag.


Which contains a rectangular hexahedron wrapped in rice paper sealed with stickers.


It's a book! A quality-looking hardback book emblazoned with a shiny faceless figure donning a cape and glasses, and there's a strip of bright orange extending from the spine of the book. There's also a 4GB SDHC memory card stuck to the front.


As you can see it's the special edition of With a Little Help by Cory Doctorow. This book is number 76 of 250. Let's have a look inside...


We appear to have a letter addressed to "Bear" from Melissa Frain from Tor (no, not the Tor project, Tor the sci-fi folk). Bear? Anyway, it's a request for the review of a manuscript. And at the back we have...


A sketch of what appears to be either a woman with a birthday cake stuck to her head, or a fairytale princess or some such variety of character.

And lastly, we have...


A personal inscription from Doctorow himself. Naturally his message alludes to the fact that he would be nothing without my help, and he's eternally grateful. (or something slightly less insane)

If you don't already know, Cory Doctorow (a Torontonian resident in the UK), amongst many other things, is a science fiction writer. This book is available for free in audio book format, on CD (MP3 or Ogg Vorbis) for a nominal fee, in several ebook formats (with a suggested donation) in paperback through on-demand publishing, or this gorgeous, unique limited edition hand-bound book. There's even the option to offer to give a book/books to institutions such as schools, prisons, hospitals etc.

So if you can get it for free, why the hell would you fork out money for it? Well for a start, it's directly re-numerating the author for the hard work they put in to make the publication you've enjoyed. That support means they'll go on to write more in future, and also affords them luxuries such as paying their energy bill and buying food to stave off death. But then the hardback book costs a fair amount. What are you getting for your money? Well, by its very nature, a limited edition item is... limited. And this one is particularly limited. The book I'm looking at is one of only 250. A personal inscription from the author isn't something you'll get with the vast majority of books (and he even spelled my name correctly). And then there's the end-papers. No-one else will have the same book as the one I have here as it has unique pieces of Cory's paper hoard that he didn't want to just throw away, so he has immortalised them forever at both ends of the book. I'm intrigued as to what this sketch is about though.

But not only that, the SD card on the front contains all the audio book files (Ogg Vorbis and MP3) every format of the ebook, and all the covers. Incidentally, the SDHC card is merely stuck into a recess in the book with some Blu-Tack, so can be easily removed and replaced.

As for the actual book itself, it says on Cory's site that these are hand-bound at the Wyvern Bindery in Clerkenwell, London, and the printing of the book is done by a family-run company called Oldacres in Hatton Garden who have been around since 1897.

The book costs the same (including P&P) regardless of where you live, except being in the UK gave me the benefit of getting it next day.

Now that the book is in my hands, I'm very pleased with my purchase and would recommend it to anyone who appreciates both good sci-fi, and beautiful, high-quality, unique well-made books. Plus the money goes directly back to the writer instead of mostly eaten away through a complicated chain of publishers, distributors and third-party sellers.

One last thing: Cory added a "feature" you don't get with most books; the opportunity to submit typo corrections with the advantage being that the typo will be fixed with a credit to yourself in the next printing of the book.

If you would like your very own copy, go get it.

Monday, 7 November 2011

Writable Common Table Expressions

There have previously been various posts on Writable Common Table Expressions (wCTEs) which demonstrate either performance gains or simple examples of how they work, but recently when giving some training, I wanted to come up with some new material to show folk that demonstrates an example of when you might wish to use it. Note that writable common table expressions are only possible in PostgreSQL, and only from version 9.1 onwards. But because 9.1 has been out for almost 2 months now, that means you can go and use this feature straight away!

So in case you don't know what a writable common table expression is, in summary it's the same as a normal common table expression, except that you can also use UPDATEs, DELETEs and INSERTs with it. If you don't know what a normal common table expression is, it effectively let's you make the results of queries available to other queries in the same statement.

What follows is a rather basic case, but the data is kept simple so as to make it easier to follow the examples.

First, we create a company payroll table:

CREATE TABLE company_payroll
(
department text PRIMARY KEY,
staff_count int NOT NULL,
staff_cost numeric(16,2) NOT NULL
);

Which we'll populate with some data:

INSERT INTO company_payroll VALUES
('development', 45, 2839281.33),
('marketing', 8, 412424.45),
('management', 4, 839432.53);

Then we want an employees table:

CREATE TABLE employees
(
id serial PRIMARY KEY,
department text REFERENCES company_payroll (department),
salary numeric(16,2),
consecutive_days_late int NOT NULL DEFAULT 0
);

Then stick some values in it:

INSERT INTO employees (department, salary, consecutive_days_late) VALUES
('development', 12834.23, 0),
('development', 37853.55, 6),
('marketing', 24488.23, 2),
('development', 91824.29, 14),
('marketing', 24829.23, 16),
('marketing', 24488.23, 2),
('management', 100293.11, 14),
('management', 40287.11, 18),
('development', 72843.22, 11);

It doesn't matter that the employees don't add up to the details provided in the company payroll. Just pretend there are many other rows, but no-one else has been late.

So we now have 2 tables looking like this:

test=# select * from company_payroll;
department | staff_count | staff_cost
-------------+-------------+------------
development | 45 | 2839281.33
marketing | 8 | 412424.45
management | 4 | 839432.53
(3 rows)

test=# select * from employees;
id | department | salary | consecutive_days_late
----+-------------+-----------+-----------------------
1 | development | 12834.23 | 0
2 | development | 37853.55 | 6
3 | marketing | 24488.23 | 2
4 | development | 91824.29 | 14
5 | marketing | 24829.23 | 16
6 | marketing | 24488.23 | 2
7 | management | 100293.11 | 14
8 | management | 40287.11 | 18
9 | development | 72843.22 | 11
(9 rows)

Okay, so we have 9 employees, but quite a few of them keep turning up late for work. Management have decided to put their foot down and fire anyone who's been late for more than 10 days in a row (and probably break a few employment laws in the process). We want to delete these employees from the employees table, then update the company payroll to reflect these changes. Then finally we want to see some before and after figures to compare the new figures against the old.

So let's present a writable common table expression to solve the problem, then we'll break it down.

WITH fired_employees AS (
DELETE FROM employees
WHERE consecutive_days_late > 10
RETURNING *),
summarise_fired_staff AS (
SELECT department, COUNT(*) AS fired_count, SUM(salary) AS salary_sum
FROM fired_employees
GROUP BY department),
updated_financials AS (
UPDATE company_payroll
SET staff_count = staff_count - fired_count,
staff_cost = staff_cost - salary_sum
FROM summarise_fired_staff
WHERE company_payroll.department = summarise_fired_staff.department
RETURNING company_payroll.*
)
SELECT
cp.department, cp.staff_count AS old_staff_count,
uf.staff_count AS new_staff_count,
cp.staff_cost AS old_staff_cost,
uf.staff_cost AS new_staff_cost
FROM company_payroll cp
INNER JOIN updated_financials uf ON uf.department = cp.department;

So let's take this one step at a time. The first thing we're doing is deleting all employees from the employees table where they've been late for more then 10 consecutive days:

DELETE FROM employees
WHERE consecutive_days_late > 10
RETURNING *

Okay, so that's simple enough. But what we've done here is put it in a WITH clause and labelled the result as fired_employees. This can now be treated like a temporary table, but one which only exists for the duration of the query. Note that we're also using the RETURNING clause of the DELETE statement to have it return all rows that were deleted by the statement. It's the output of this which is now represented by the fired_employees table expression name. We can now use this for other purposes, and we will. So fired_employees now contains:

id | department | salary | consecutive_days_late
----+-------------+-----------+-----------------------
4 | development | 91824.29 | 14
5 | marketing | 24829.23 | 16
7 | management | 100293.11 | 14
8 | management | 40287.11 | 18
9 | development | 72843.22 | 11
(5 rows)

Next we summarise how many people we've fired from each department, and how much they cost in total per department:

SELECT department, COUNT(*) AS fired_count, SUM(salary) AS salary_sum
FROM fired_employees
GROUP BY department

Again, very straightforward. Note that we're referring to the fired_employees expression which we can treat like a regular table, and the output of this has been named summarise_fired_staff. This contains:

department | fired_count | salary_sum
-------------+-------------+------------
development | 2 | 164667.51
management | 2 | 140580.22
marketing | 1 | 24829.23
(3 rows)

As you can see, we fired 2 developers worth 164667.51 in whatever currency you want to pretend this represents. We've also fired 2 from management and 1 from marketing.

Now we want to update the figures we maintain in the company payroll table with this information:

UPDATE company_payroll
SET staff_count = staff_count - fired_count,
staff_cost = staff_cost - salary_sum
FROM summarise_fired_staff
WHERE company_payroll.department = summarise_fired_staff.department
RETURNING company_payroll.*

Here we're joining the company_payroll to the summarise_fired_staff expression based on the department name. The update involves decrementing the staff_count by the fired_count returned in the summarise_fired_staff result, and the same for the staff_cost, which is decremented by the salary_sum. Now we want to return the updated rows from this action, so we use "RETURNING company_payroll.*". And the output of this part of the query we've labelled updated_financials. As you can see, it's similar to how bash commands are used; you pass the output from one to another, and so on. The result of each one gets passed to the next. And now updated_financials contains:

department | staff_count | staff_cost
-------------+-------------+------------
development | 43 | 2674613.82
management | 2 | 698852.31
marketing | 7 | 387595.22
(3 rows)

So finally we want to see a side-by-side comparison of before and after figures:

SELECT
cp.department, cp.staff_count AS previous_staff_count,
uf.staff_count AS new_staff_count,
cp.staff_cost AS previous_staff_cost,
uf.staff_cost AS new_staff_cost
FROM company_payroll cp
INNER JOIN updated_financials uf ON uf.department = cp.department

So we've joined the still yet-to-be-actually-updated company_payroll table with the updated version (which we named updated_financials) based on department name. We then select the department, followed by the original staff_count from the company_payroll table (labelling it previous_staff_count for clarity), and then the updated_financials own staff_count colum (labelled new_staff_count). We also do the same for the staff cost.

So running all of this in one go returns the following:

department | old_staff_count | new_staff_count | old_staff_cost | new_staff_cost
-------------+-----------------+-----------------+----------------+----------------
development | 45 | 43 | 2839281.33 | 2674613.82
management | 4 | 2 | 839432.53 | 698852.31
marketing | 8 | 7 | 412424.45 | 387595.22
(3 rows)

If you're curious to know what the explain plan for this query is, it's as follows:

QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=38.33..39.40 rows=3 width=80) (actual time=0.256..0.258 rows=3 loops=1)
Hash Cond: (cp.department = uf.department)
CTE fired_employees
-> Delete on employees (cost=0.00..21.75 rows=313 width=6) (actual time=0.059..0.066 rows=4 loops=1)
-> Seq Scan on employees (cost=0.00..21.75 rows=313 width=6) (actual time=0.022..0.023 rows=4 loops=1)
Filter: (consecutive_days_late > 10)
Rows Removed by Filter: 12
CTE summarise_fired_staff
-> HashAggregate (cost=8.61..10.61 rows=200 width=48) (actual time=0.107..0.108 rows=3 loops=1)
-> CTE Scan on fired_employees (cost=0.00..6.26 rows=313 width=48) (actual time=0.062..0.071 rows=4 loops=1)
CTE updated_financials
-> Update on company_payroll (cost=1.07..5.88 rows=3 width=198) (actual time=0.184..0.198 rows=3 loops=1)
-> Hash Join (cost=1.07..5.88 rows=3 width=198) (actual time=0.159..0.167 rows=3 loops=1)
Hash Cond: (summarise_fired_staff.department = company_payroll.department)
-> CTE Scan on summarise_fired_staff (cost=0.00..4.00 rows=200 width=168) (actual time=0.127..0.131 rows=3 loops=1)
-> Hash (cost=1.03..1.03 rows=3 width=62) (actual time=0.008..0.008 rows=3 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on company_payroll (cost=0.00..1.03 rows=3 width=62) (actual time=0.002..0.003 rows=3 loops=1)
-> Seq Scan on company_payroll cp (cost=0.00..1.03 rows=3 width=56) (actual time=0.029..0.030 rows=3 loops=1)
-> Hash (cost=0.06..0.06 rows=3 width=56) (actual time=0.205..0.205 rows=3 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> CTE Scan on updated_financials uf (cost=0.00..0.06 rows=3 width=56) (actual time=0.188..0.203 rows=3 loops=1)
Total runtime: 0.514 ms
(23 rows)

You could also easily add in delta (or diff) columns to show how much they've changed by, because you have 2 sets of columns to reference and do a straight-forward calculation, then label those columns accordingly.

So what initially appears as a complicated behemoth of a statement is actually fairly simple once you break it down. And while the example shown isn't necessarily realistic, it should hopefully demonstrate its potential applications.

And you can try out this example right away if you're running PostgreSQL 9.1. Just run the CREATE TABLE and INSERT statements above, then run the big writable common table expression query and see the results for yourself. Note that because the wCTE query actually makes changes, running it more than once will do nothing since the employees that were fired were removed by the query the first time it was run.

You can read other blog entries on wCTEs at these links:

Thursday, 1 September 2011

The first PostgreSQL Conference Europe talks!

The first round of PostgreSQL Conference Europe talks have now been selected and already it's shaping up to be a great conference if these are any to go by. You can see the list so far at http://www.postgresql.eu/events/sessions/pgconfeu2011/.

If we take a brief look at a few of the talks, you'll see that you really don't want to miss this year's line-up.

Firstly we have a talk by PostgreSQL veteran and core team member Bruce Momjian who will be "Explaining the Postgres Query Optimizer". This will enlighten attendees as to how PostgreSQL makes its decisions in response to queries which is important to understand in order to know how best to ask PostgreSQL what you want. Bruce gave a very popular MVCC talk last year, and no doubt this one will be one to watch.

Also, Michael Meskes's talk entitled "Mission impossible? Can I replace my most important databases with PostgreSQL?" won't be a brief "yes", but will elaborate on Michael's experience with over 12 years of using open source in business, and how to make migrations work. He'll also provide some real life examples of the benefits which have been realised over the years.

And you'll also get the chance to attend the "Synchronous Replication and Durability Tuning" talk by Simon Riggs, who developed the synchronous replication functionality, and Greg Smith, who wrote the book on PostgreSQL performance. No really, he did. In this talk Simon and Greg will go through how synchronous replication works in PostgreSQL 9.1, what considerations should be taken into account, and typical performance users of this feature can expect. Then they will cover how to go about improving performance for this particular set up.

We don't just have talks though, there are also training sessions available, and in addition to the ones previously announced we now also have "Streaming replication, hot standby and synchronous replication" by Redpill Linpro's own Magnus Hagander. There's also "Hands-On with PostGIS" by Vincent Picavet, a GIS specialist who's providing training on behalf of Dalibo. Go check out the complete list at http://2011.pgconf.eu/training/.

These talks shown so far are just the start. At the time of writing this we have confirmed about 15 talks (probably more as you read this), but there will be many many more, so keep your eye on the conference website and subscribe to the conference Twitter feed for the latest updates. And get don't forget to make your booking for training sessions in advance as there are a limited number of allocations for each class.