Removing exactly one physical volume from an ext4 filesystem

While there appear to be many webpages explaining how to resize/shrink a volume, I haven’t seen one that resizes/shrink the filesystem to exactly the right size, and then resizes/reduces the logical volume to match.  The accepted approach appears to be to resize the filesystem to definitely smaller than the logical volume will be, then reducing the logical volume, then resizing/growing the filesystem to match the logical volume.

Here I will attempt to remove /dev/sdc from /dev/vg_dev/lv_home without excessive resizing.  I will be taking advantage of lvreduce‘s --resizefs, which may not be available on your system, so please check before proceeding.

1. Run pvdisplay and look at how many Allocated PE (physical extents) you will remove.  In my case, 715396.
2. Run lvreduce with -l and --resizefs to reduce the logical volume and resize the filesystem at the same time.
3. Run vgreduce to remove the physical volume from the volume group.
4. Run pvremove so the devices is not recognized as a physical volume.

pvdisplay
lvreduce -l -715396 --resizefs /dev/vg_dev/lv_home
vgreduce vg_dev /dev/sdc
pvremove /dev/sdc

Dependency Injection Design Pattern in Java: Example

Dependency Injection Design Pattern

Program to an ‘interface’, not an ‘implementation’.

This is the premise of dependency injection. It goes one step further to allow libraries that use the interface library to not ‘depend’ on the implementation library even though they depend on the interface library. That is, the implementation can be ‘injected’ at run time. This isn’t too hard to do without dependency injection.  The implementation library simply constructs an instance of the implementation and passes it to some method or sets some field in the dependent library.  Or you could have another library that depends on the dependent library and the implementation library. But what if you want to construct the instance in the interface library?  You could use introspection to locate the implementation classes and construct them.  This is one way to implement dependency injection.

In Java

As mentioned before, you could use introspection to locate the implementation classes and construct them without depending on the implementation library.  Writing this yourself with isAssignableFrom isn’t too bad but you’ll need to load every class into your class loader.  You could depend on a library that understand bytecode, but if you’re going to do that, you might as well use a dependency injection framework.

Example

In this example, we’ll use HK2 2.1 (a dependency injection framework).  Why HK2?  Because I’m using Jersey 2 and Jersey 2 uses HK2.  Why 2.1?  Well, because I’m using an old Jersey 2 and it uses 2.1.  Also, there is plenty of documentation available for 2.2 and above (just my luck, right?) but not enough for 2.1.

The interface library

Since I’m using maven for dependency management, here’s the maven dependency you’ll need to add (if it’s not already included by another dependency) for the annotations you’ll use in this example.

<dependency>
    <groupId>org.glassfish.hk2</groupId>
    <artifactId>hk2-api</artifactId>
    <version>${hk2.version}</version>
</dependency>

The interface

There’s not much to do here. Just add an annotation.

import org.jvnet.hk2.annotations.Contract;
@Contract
public interface InterfaceContract {
}

The implementation library

There should be a dependency on your interface library, which will already bring in the dependency required for the annotation, but HK2 needs a file that describes each service in the library. This file should be in your jar file at META-INF/hk2-locator/default and will be generated in the right location by a maven plugin with the following code. You can find more information on generating inhabitants.

            <plugin>
                <groupId>org.glassfish.hk2</groupId>
                <artifactId>hk2-inhabitant-generator</artifactId>
                <version>${hk2.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>generateInhabitants</goal> <!-- generate-inhabitants in hk2 2.2+ -->
                        </goals>
                    </execution>
                </executions>
            </plugin>

The implementation class

There’s not much to do here. Just add an annotation. You could get fancy with the annotations depending on what you want to do, but I’m trying to keep it basic.

import org.jvnet.hk2.annotations.Service;
@Service
public class ImplementationService implements InterfaceContract {
}

The constructing library

Another dependency here, so you can tell HK2 to give you an instance of InterfaceContract without depending on the implementation library.

        <dependency>
            <groupId>org.glassfish.hk2</groupId>
            <artifactId>core</artifactId>
            <version>${hk2.version}</version>
        </dependency>

So I just wanted to have one ServiceLocator in my library. Here’s a way to create an anonymous locator and populate it with all the services in the $CLASSPATH. It scans jar files for META-INF/hk2-locator/default so it doesn’t actually have to load your class until needed.

ServiceLocatorFactory is a singleton, so you don’t need to worry about creating them.

I created the ServiceLocator as a constant and populated it statically.  You can give your locator a name, and then use ServiceLocatorFactory.find(name) to get it later. This way, you don’t need to keep your locator in a field and you don’t need to keep creating/populating locators.

import org.glassfish.hk2.api.ServiceLocator;
import org.glassfish.hk2.api.ServiceLocatorFactory;
import org.glassfish.hk2.bootstrap.HK2Populator;

    private static final ServiceLocator LOCATOR = ServiceLocatorFactory.getInstance().create(null);
    static {
        try {
            HK2Populator.populate(LOCATOR);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

The constructing method

Almost as easy as calling new ImplementationService(). Will return null if it can’t find any services. If your service is annotated with @javax.inject.Singleton, it will create the first instance, and then return that during later calls. That’s the basic way to get an instance of a service, but there’s plenty more.

        LOCATOR.getService(InterfaceContract.class);

What about @Inject?

You may have noticed that I haven’t mentioned the annotation @javax.inject.Inject like many other webpages do. It can appear on a field or method (including constructor) and will automatically use the ServiceLocator to fill in the field or method parameter(s) with a service. The ServiceLocator used to create the service will be the same one used to create the containing instance, so this means you have to use HK2 to create the containing instance, either by using the basic way to get an instance of a service, or unmanaged creation/injection.

I haven’t used @Inject in the example above because it’s not needed for dependency injection in HK2. (Can’t comment on other frameworks.) You can get pretty fancy with specifying what you want to @Inject.

A more concrete example

Ok, so the above all sounds very abstract, so let’s have a more concrete example. Let’s say you are asked to create a logging framework. Your logging framework should not depend on a specific destination for the log messages, but rather, should allow the implementation to be decided at runtime. Your framework consists of two library jar files: an interface library log-interface.jar and an example implementation log-implementation-console.jar.

The interface in Log.java in log-interface.jar looks like:

import org.jvnet.hk2.annotations.Contract;
@Contract
public interface Log {
  /**
   * Logs a message
   *
   * @param message the message to log.
   */
  void log(String message);
}

The implementation in ConsoleLog.java in log-implementation-console.jar looks like:

import org.jvnet.hk2.annotations.Service;
@Singleton
@Service
public class ConsoleLog implements Log {
  @Override
  void log(String message) {
    System.out.println(message);
  }
}

Now any code in any library (including log-interface.jar) can use ConsoleLog without knowing about it:

Log LOG = LOCATOR.getService(Log.class);
LOG.log("Hello, world!");

You just need to make sure that ConsoleLog.class is in your $CLASSPATH at runtime, possibly by including log-implementation-console.jar in your WAR file.

Workaround transient I/O errors

So apparently some older linux kernels have issues during times of high I/O and cause drives to be stuck reporting I/O errors until a reboot. Apparently this is a more common issue on virtual machines. If this doesn’t happen to the drive containing your OS, you’re lucky because you can recover from this without rebooting. The commands below are specific to my case:

• Debian squeeze 6.0.10
• Linux 2.6.32
• LVM physical volume /dev/sdc had the error
• There are other LVM physical volumes on the system
• Physical machine (not virtual machine)

sudo umount /dev/shm/mnt/                                       # stop using the mountpoint
sudo lvremove /dev/woodcock_massive/woodcock_storage            # i/o errors
sudo vgremove /dev/woodcock_massive                             # i/o errors
sudo pvremove /dev/sdc1                                         # i/o errors
sudo pvscan                                                     # i/o errors
echo 1 | sudo tee /sys/block/sdc/device/delete                  # remove sdc and stop all these i/o errors
sudo dmsetup remove /dev/dm-6                                   # remove this lingering device mapper "device" that causes i/o errors
ls -l /dev/sdc                                                  # ???  where's my drive?  we really did remove it.
echo "0 0 0" | sudo tee /sys/class/scsi_host/host4/scan         # scan, find, and readd /dev/sdc
sudo vgchange -ay woodcock_massive                              # make the volume group available
sudo mount /dev/woodcock_massive/woodcock_storage /dev/shm/mnt/ # go back to the state we were in

git status sorted by last modified timestamp

For those of us that keep a heap of local changes (both new and changed files), here’s a command that will show you unstaged files, sorted by time:

ls -lrtd $(git status --porcelain | grep '^.[?M]' | sed 's/^.. //')

I find this particularly useful to commit what I worked on that day.

If you know how to make a git alias for this, please let me know.

umount mountpoint and all sub-mountpoints

A bash function to show the most reliable way I’ve found:

unmount() {
  sudo umount -l $(cut -f5 -d ' ' /proc/self/mountinfo | egrep "^$1(/|$)" |
                   sort -r | uniq | xargs)
}

• What’s wrong with /proc/mounts?
  • If you have a “device” that’s been deleted, it makes the output harder to parse.
• Why not just grep ^"$1"?
  • You may have something mounted at "$1"foo.  (dirname has the prefix of $1.)
• Why uniq?
  • You may be have mounted onto the same directory twice at some point, which can confuse /proc/mounts.
• Why not just sudo umount -l "$1"?
  • That doesn’t unmount the sub-mounts.

Moved to wordpress.com

Not that there are many on CSE who host their blog using wordpress, but for those that have, to migrate:

1. export from CSE (e.g., https://cgi.cse.unsw.edu.au/~jayen/blog/wp-admin/export.php)
2. import onto wordpress.com (e.g., https://jayenashar.wordpress.com/wp-admin/admin.php?import=wordpress)
3. choose an appropriate theme (if you were using a custom theme on CSE that you can’t use on wordpress.com)
4. fix up your widgets
5. move any disallowed media to a third-party host (e.g., dropbox)

javascript every language (and country) in every other language

Thank you java for supplying this information:

Full lookup:

codetonames.js

Some duplicates removed:

codetonames.js

retry function for bash

retry() {
    local tries_left=$1
    shift
    local sleep_time=$1
    shift

    until "$@"; do
        ((tries_left--))
        if [[ $tries_left -le 0 ]]; then
            return 275
        fi
        sleep $sleep_time
    done
}

Example:

retry 12 5 ssh test_host true

logging stdout and stderr from within a shell script while still outputting to the console

If you want to send the output of a shell script to files, it’s pretty easy:

./script.sh 1> out.txt 2> err.txt

If you want to do it within a script, also easy:

exec 1> out.txt
exec 2> err.txt

If you want to send it to the console and files, it’s a bit more tricky:

{ ./script.sh 2>&3 | tee out.txt; } 3>&1 1>&2 | tee err.txt

(Note that 1 now contains the original stderr, and the original stdout is only accessible inside the curly braces {}.)

But if you want to do this within a script, I’d suggest employing the use of named pipes (FIFOs):

mkfifo /tmp/$$.out.pipe
tee out.txt < /tmp/$$.out.pipe &
exec 1> /tmp/$$.out.pipe

mkfifo /tmp/$$.err.pipe
tee err.txt < /tmp/$$.err.pipe >&2 &
exec 2> /tmp/$$.err.pipe

EDIT: This is much easier:

exec > >(tee /tmp/stdout.txt) 2> >(tee /tmp/stderr.txt >&2)

YAGBM (Yet Another Git Branching Model)

Shamelessly plagiarized from http://nvie.com/posts/a-successful-git-branching-model/ (but adapted to the model I like to use).

Exactly what to develop in which branches is a puzzling problem, especially for developers new to branching, large teams, and developers with regular releases.

In this post I present the development model that I try to introduce for all of my projects (both at work and private). I’ve been meaning to write about it for a while now, but I’ve never really found the time to do so thoroughly, until now, but only because I shamelessly plagiarize so much. I also won’t talk about any of the projects’ details, merely about the branching strategy and release management.

It focuses around Git as the tool for the versioning of all of our source code.

Why git?

I’m sure any code repository system that properly supports branching will be fine. I’m used to git and I haven’t worked on a project that uses mercurial or darcs, but they’re probably fine as well. RCS/SCCS/CVS/SVN do not do branching nicely, if at all. A decentralized system like git/mercurial/darcs is not necessary, but you should be familiar with how to get others’ changes for whichever tool you use.

Enough about the tools, let’s head onto the development model. The model that I’m going to present here is essentially no more than a set of procedures that every team member has to follow in order to come to a managed software development process.

The main branch

At the core, the development model is greatly inspired by existing models out there. The central repository (origin) holds one main branch with an infinite lifetime, master.

The master branch at origin should be familiar to every Git user.

We consider origin/master to be the main branch where the source code of HEAD always reflects a state with the latest delivered development changes for the next release. Some would call this the “integration branch”. This is where any automatic nightly builds are built from.

When the source code in the master branch reaches a stable point and is ready to be released, all of the changes should be branched off somehow and then tagged with a release number. How this is done in detail will be discussed further on.

Therefore, each time when changes are merged back into master, this will go into the next minor release.

---

Supporting branches

Next to the main branch master, our development model uses a variety of supporting branches to aid parallel development between team members, ease tracking of features, prepare for production releases and to assist in quickly fixing live production problems. Unlike the main branches, these branches always have a limited life time, since they will be removed eventually.

The different types of branches we may use are:

• Feature branches
• Release branches

Each of these branches have a specific purpose and are bound to strict rules as to which branches may be their originating branch and which branches must be their merge targets. We will walk through them in a minute.

By no means are these branches “special” from a technical perspective. The branch types are categorized by how we use them. They are of course plain old Git branches.

Feature branches

May branch off from: origin/master
Must merge back into: origin/master
Branch naming convention: anything except master or release-*

Feature branches (or sometimes called topic branches) are used to develop new features for the upcoming or a distant future release. When starting development of a feature, the target release in which this feature will be incorporated may well be unknown at that point. The essence of a feature branch is that it exists as long as the feature is in development, but will eventually be merged back into master (to definitely add the new feature to the upcoming release) or discarded (in case of a disappointing experiment).

Creating a feature branch

When starting work on a new feature, branch off from the master branch.

$ git checkout -b myfeature origin/master
Switched to a new branch "myfeature"

Incorporating a finished feature onto master

Finished features may be merged into the master branch to definitely add them to the upcoming release:

$ git merge --no-ff origin/master
Updating ea1b82a..05e9557 (Summary of changes)
$ git push origin myfeature:master :myfeature
$ git branch -d myfeature
Deleted branch myfeature (was 05e9557).

The --no-ff flag causes the merge to always create a new commit object, even if the merge could be performed with a fast-forward. This avoids losing information about the historical existence of a feature branch and groups together all commits that together added the feature. Compare:

In the latter case, it is impossible to see from the Git history which of the commit objects together have implemented a feature—you would have to manually read all the log messages. Identifying a whole feature (i.e. a group of commits), is a true headache in the latter situation, whereas it is easier to do if the --no-ff flag was used.

Yes, it will create a few more (empty) commit objects, but the gain is much bigger that that cost.

You can make --no-ff the default behaviour of git merge by running git config merge.ff false

The fancy push command pushes myfeature (which has all the commits from origin/master and myfeature) to origin/master and also deletes origin/myfeature, if it existed.

Release branches

May branch off from: master
Must merge back into: master
Branch naming convention: release-*

Release branches support preparation of a new production release. They allow for last-minute dotting of i’s and crossing t’s. Furthermore, they allow for minor bug fixes and supporting multiple releases simultaneously. By doing all of this work on a release branch, the master branch is cleared to receive features for the next big release.

The key moment to branch off a new release branch from master is when master (almost) reflects the desired state of the new release. At least all features that are targeted for the release-to-be-built must be merged in to master at this point in time. All features targeted at future releases must not—they must wait until after the release branch is branched off.

It is exactly at the start of a release branch that the upcoming release gets assigned a version number—not any earlier. Up until that moment, the master branch reflected changes for the “next release”, but it is unclear whether that “next release” will eventually become 0.3 or 1.0, until the release branch is started. That decision is made on the start of the release branch and is carried out by the project’s rules on version number bumping.

Creating a release branch

Release branches are created from the master branch. For example, say version 1.1 is the current production release and we have a big release coming up. The state of master is ready for the “next release” and we have decided that this will become version 1.2 (rather than 2.0). So we branch off and give the release branch a name reflecting the new version number:

$ git checkout -b release-1.2 master
Switched to a new branch "release-1.2"
$ ./bump-version.sh 1.2.0
Files modified successfully, version bumped to 1.2.0.
$ git commit -a -m "Bumped version number to 1.2.0"
[release-1.2 74d9424] Bumped version number to 1.2.0
1 files changed, 1 insertions(+), 1 deletions(-)

After creating a new branch and switching to it, we bump the version number. Here, bump-version.sh is a fictional shell script that changes some files in the working copy to reflect the new version. (This can of course be a manual change—the point being that some files change.) Then, the bumped version number is committed.

This new branch may exist there for a while, until the release is no longer supported. During that time, bug fixes may be applied in this branch (rather than on the master branch). You could repeatedly bump the version number for bug fix releases (1.2.1, 1.2.2, etc.) Adding large new features here is strictly prohibited. They must be merged into master, and therefore, wait for the next big release.

Rolling out a release branch

Each time we want to roll-out, we tag the code the release was built from. We tend to be very strict at this, so that theoretically, we could use a Git hook script to automatically build and roll-out our software to our production servers everytime there was a tag on release-*.

When the state of the release branch is ready to become a real release, some actions need to be carried out. First, the release branch must be tagged for easy future reference to this historical version. Secondly, the changes made on the release branch need to be merged into the branches of other supported releases and master, so that future releases also contain these bug fixes.

The first step in Git:

$ git tag -a 1.2.0

The release is now done, and tagged for future reference.
You might as well want to use the -s or -u <key> flags to sign your tag cryptographically.

To keep the changes made in the release branch, we need to merge those back into master, though. In Git:

$ git checkout master
Branch master set up to track remote branch master from origin.
Switched to a new branch 'master'
$ git merge --no-ff release-1.2
Merge made by the 'recursive' strategy.
(Summary of changes)
$ git push origin master release-1.2

This step may well lead to a merge conflict (probably even, since we have changed the version number). If so, fix it and commit. Alternatively, you can cherry-pick the commits you want. This may be more painful, but won’t put unwanted version-bump commits in the other release branches:

$ git checkout master
Branch master set up to track remote branch master from origin.
Switched to a new branch ‘master’
$ git cherry-pick release-1.2.0..release-1.2.1^
$ git push origin master release-1.2

In this example, we cherry-picked everything after the tag release-1.2.0 and before the tag release-1.2.1. Presumably the unwanted commits are the tagged ones.

Now we are really done and the local copy of the master branch may be removed, since we don’t need it anymore:

$ git checkout release-1.2
$ git branch -d master
Deleted branch master (was ff452fe).

Hotfixes

May tag in: release-*
Must merge/cherry-pick into: release-* and master
Tag naming convention: release-x.y.z

Hotfixes are done in release branches as they also are meant to prepare for a new production release, albeit unplanned. They arise from the necessity to act immediately upon an undesired state of a live production version. When a critical bug in a production version must be resolved immediately, a hotfix commit may be tagged in the corresponding release branch that marks the production version.

The essence is that work of team members (on the master branch) can continue, while another person is preparing a quick production fix.

Creating the hotfix commit

Hotfixes are created from the release-* branch. For example, say version 1.2.0 is the current production release running live and causing troubles due to a severe bug. But changes on master are yet unstable. We may then check out the problematic branch and start fixing the problem:

$ git checkout -b release-1.2
Branch release-1.2 set up to track remote branch release-1.2 from origin.
Switched to a new branch 'release-1.2'

Then, fix the bug and commit the fix in one or more separate commits.

$ git commit -m "Fixed severe production problem"
[release-1.2 abbe5d6] Fixed severe production problem
5 files changed, 32 insertions(+), 17 deletions(-)

Rolling out a hotfix

When finished, the bugfix needs to be merged back into master, but also needs to be merged back into the other release branches, in order to safeguard that the bugfix is included in all supported releases as well. This is completely similar to how release branches are rolled out.

First, tag the release.

$ ./bump-version.sh 1.2.1
Files modified successfully, version bumped to 1.2.1.
$ git commit -a -m "Bumped version number to 1.2.1"
[release-1.2 41e61bb] Bumped version number to 1.2.1
1 files changed, 1 insertions(+), 1 deletions(-)
$ git tag -a 1.2.1

Don’t forget to bump the version number before tagging!
You might as well want to use the -s or -u <key> flags to sign your tag cryptographically.

Next, include the bugfix in master, too:

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff release-1.2
Merge made by the 'recursive' strategy.
(Summary of changes)

The one exception to the rule here is that, when another release branch is currently supported, the hotfix changes need to be merged into that release branch, in addition to master. Back-merging the bugfix into the master branch will eventually result in the bugfix being merged into future release branches too, when the releases are ready, but doesn’t help for past releases.

Summary

While there is nothing really shocking new to this branching model, the “big picture” figure that this post began can turn out to be tremendously useful in your projects. It forms an elegant mental model that is easy to comprehend and allows team members to develop a shared understanding of the branching and releasing processes.

A high-quality PDF version of the figure is provided here. Go ahead and hang it on the wall for quick reference at any time.

And for anyone who wants it: here’s the gitflow-model.src.svgz of the main diagram image (Inkscape).

Git-branching-model.pdf

Feel free to add your comments!

If you want to get in touch, I’m @jayenashar on Twitter.