Taking advantage of Apache FileUpload and opencsv
In the previous post we added support for reports to Serendipity by taking advantage of the JasperReports reporting engine and the visual report designer, iReport. In this post, we're going to add support for importing CSV files into Serendipity by taking advantage of Apache FileUpload and opencsv.
In this post, we'll:
- Install Apache FileUpload
- Install opencsv
- Import a CSV file
- Test the application in development mode
Note: You can download the sample application from the CRMdipity (Se-r-en-dipity) project's download page.
1. Install Apache FileUpload
Download and then unzip Apache FileUpload. I downloaded:
- commons-fileupload-1.2.1-bin.zip - Release 1.2.1 of Apache FileUpload
And add the following library to the project (e.g. copy them into the project's war/WEB-INF/lib directory):
- commons-fileupload-1.2.1.jar (from the lib directory)
Add the library to your build path by right-clicking on the jar file and selecting "Build Path -> Add to Build Path".
2. Install opencsv
Download and then unzip opencsv. I downloaded:
- opencsv-2.2-src-with-libs.tar.gz - Release 2.2 of opencsv
And add the following library to the project (e.g. copy them into the project's war/WEB-INF/lib directory):
- opencsv-2.2.jar (from the deploy directory)
Add the library to your build path by right-clicking on the jar file and selecting "Build Path -> Add to Build Path".
3. Import a CSV file
We can import a CSV file by clicking the New button on Serendiptiy's Main page:
First, we need to register a handler for the New button in the nested Imports View:
Note: The initToolBar method is called by the Imports View's bindCustomUiHandler method.
...
protected void initToolBar() {
getToolBar().addButton(ToolBar.NEW_IMPORT_BUTTON, Serendipity.getConstants().newButton(),
Serendipity.getConstants().newButtonTooltip(), new ClickHandler() {
public void onClick(ClickEvent event) {
if (getUiHandlers() != null) {
getUiHandlers().onNewButtonClicked();
}
}
});
...
}
We also we need to define a UI handler in the nested Imports Presenter:
...
private static final String HOST_FILENAME = "FileUpload.html";
private static final String NAME = "_blank";
private static final String FEATURES = "width=360, height=280, location=no, resizable=no";
@Override
public void onNewButtonClicked() {
StringBuilder url = new StringBuilder();
url.append(HOST_FILENAME);
Window.open(Serendipity.getRelativeURL(url.toString()), NAME, FEATURES);
}
Take a look at the Imports Presenter's onNewButtonClicked method and you will notice that it calls Window.open. This means that Serendipity needs to be able to manage multiple host pages and thanks to GWT Multipage, this can be acheived by simply adding an annotation to the FileUpload entry point class:
...
@UrlPatternEntryPoint(value = "(FileUpload.html)?((&|\\\\?)locale=(en|de))?")
public class FileUploadEntryPoint extends MultiPageEntryPoint {
protected void revealCurrentPlace(String page) {
if (page.equals(NameTokens.fileUploadPage)) {
PlaceRequest placeRequest = new PlaceRequest(NameTokens.fileUploadPage);
getSerendipityGinjector().getPlaceManager().revealPlace(placeRequest);
} else {
Log.debug("Page name token: " + page);
}
}
}
Now, when you click the New button on the Imports ToolBar, the Upload Data File window will be displayed:
The Upload Data File window allows you to choose the Record type (e.g. Account), the Data map (e.g. Automatic) and the name of the file to upload. Let's take a look at the FileUploadPage View:
...
public class FileUploadPageView extends ViewWithUiHandlers<FileUploadUiHandlers> implements
FileUploadPagePresenter.MyView {
private static final String DEFAULT_MARGIN = "0px";
private VLayout panel;
private VLayout northLayout;
private VLayout southLayout;
private DynamicForm uploadForm;
private UploadItem uploadItem;
private ComboBoxItem recordType;
private ComboBoxItem mapName;
@Inject
public FileUploadPageView() {
// get rid of scroll bars, and clear out the window's built-in margin,
// because we want to take advantage of the entire client area
Window.enableScrolling(false);
Window.setMargin(DEFAULT_MARGIN);
// initialise the main layout container
panel = new VLayout();
panel.setWidth100();
panel.setHeight100();
// initialise the North layout container
northLayout = new VLayout();
northLayout.setWidth100();
northLayout.setHeight(HEADER_HEIGHT);
// initialise the South layout container
southLayout = new VLayout();
southLayout.setWidth100();
southLayout.setHeight100();
// add the nested layout containers to the main layout containers
northLayout.addMember(initHeader());
southLayout.addMember(initBody());
southLayout.addMember(initFooter());
// add the North and South layout containers to the main layout container
panel.addMember(northLayout);
panel.addMember(southLayout);
// set the browser window's title
Window.setTitle("Upload Data File");
}
@Override
public Widget asWidget() {
return panel;
}
private static final int HEADER_HEIGHT = 58;
private static final String NAME = "Select a Data File to Upload";
private static final String DESCRIPTION = "Select a data file to be imported into Serendipity.";
private VLayout initHeader() {
// initialise the Header layout container
VLayout header = new VLayout();
header.setStyleName("crm-Wizard-Header");
header.setWidth100();
header.setHeight(HEADER_HEIGHT);
HLayout line1 = new HLayout();
line1.setWidth100();
line1.setHeight100();
HLayout line2 = new HLayout();
line2.setWidth100();
line2.setHeight100();
// initialise the Name label
Label name = new Label();
name.setStyleName("crm-Wizard-Name");
name.setContents(NAME);
name.setWidth100();
// initialise the Description label
Label description = new Label();
description.setStyleName("crm-Wizard-Description");
description.setContents(DESCRIPTION);
description.setWidth100();
// add the labels to the nested layout containers
line1.addMember(name);
line2.addMember(description);
// add the North and South layout containers to the main layout container
header.addMember(line1);
header.addMember(line2);
return header;
}
private static final String DEFAULT_FILE_UPLOAD_SERVICE_PATH = "upload";
private static final String RECORD_TYPE = "recordType";
private static final String MAP_NAME = "mapName";
private static final String TARGET="uploadTarget";
private VLayout initBody() {
// initialise the Footer layout container
VLayout body = new VLayout();
body.setStyleName("crm-Wizard-Body");
body.setWidth100();
body.setHeight100();
// initialise the form
uploadForm = new DynamicForm();
uploadForm.setWidth100();
uploadForm.setMargin(8);
uploadForm.setNumCols(2);
uploadForm.setCellPadding(2);
uploadForm.setWrapItemTitles(false);
// no ":" after the field name
uploadForm.setTitleSuffix(" ");
uploadForm.setRequiredTitleSuffix(" ");
// initialise the hidden frame
NamedFrame frame = new NamedFrame(TARGET);
frame.setWidth("1px");
frame.setHeight("1px");
frame.setVisible(false);
uploadForm.setEncoding(Encoding.MULTIPART);
uploadForm.setMethod(FormMethod.POST);
// set the (hidden) form target
uploadForm.setTarget(TARGET);
StringBuilder url = new StringBuilder();
url.append(DEFAULT_FILE_UPLOAD_SERVICE_PATH);
uploadForm.setAction(FileUploadEntryPoint.getRelativeURL(url.toString()));
// initialise the Record type field
recordType = new ComboBoxItem(RECORD_TYPE, "Record type");
recordType.setName(RECORD_TYPE);
recordType.setType("comboBox");
recordType.setValueMap("Account", "Contact");
recordType.setDefaultToFirstOption(true);
recordType.setSelectOnFocus(true);
// initialise the Map field
mapName = new ComboBoxItem(MAP_NAME, "Data map");
mapName.setName(MAP_NAME);
mapName.setType("comboBox");
mapName.setValueMap("Automatic", "Create new map");
mapName.setDefaultToFirstOption(true);
// initialise the File name field
uploadItem = new UploadItem("filename");
uploadItem.setName("filename");
uploadItem.setTitle("File name");
uploadItem.setWidth(280);
// set the fields into the form
uploadForm.setFields(recordType, mapName, uploadItem);
// add the Upload Form and the (hidden) Frame to the main layout container
body.addMember(uploadForm);
body.addMember(frame);
return body;
}
private static final int FOOTER_HEIGHT = 48;
private VLayout initFooter() {
// initialise the Footer layout container
VLayout footer = new VLayout();
footer.setStyleName("crm-Wizard-Footer");
footer.setWidth100();
footer.setHeight(FOOTER_HEIGHT);
HLayout hLayout = new HLayout();
hLayout.setWidth100();
hLayout.setHeight(FOOTER_HEIGHT);
// initialise the OK button
IButton okButton = new IButton("OK");
okButton.setShowRollOver(true);
okButton.setShowDisabled(true);
okButton.setShowDown(true);
okButton.addClickHandler(new ClickHandler() {
public void onClick(ClickEvent event) {
Object obj = uploadItem.getDisplayValue();
if (obj != null) {
uploadForm.submitForm();
if (getUiHandlers() != null) {
getUiHandlers().onOkButtonClicked();
}
} else {
SC.say("Please select a file.");
}
}
});
// initialise the Cancel button
IButton cancelButton = new IButton("Cancel");
cancelButton.setShowRollOver(true);
cancelButton.setShowDisabled(true);
cancelButton.setShowDown(true);
cancelButton.addClickHandler(new ClickHandler() {
public void onClick(ClickEvent event) {
if (getUiHandlers() != null) {
getUiHandlers().onCancelButtonClicked();
}
}});
// layout the OK and Cancel buttons
hLayout.setLayoutMargin(8);
hLayout.addMember(new LayoutSpacer());
hLayout.addMember(okButton);
LayoutSpacer padding = new LayoutSpacer();
padding.setWidth(8);
hLayout.addMember(padding);
hLayout.addMember(cancelButton);
// add the nested layout container to the main layout container
footer.addMember(hLayout);
return footer;
}
}
And you'll notice that most of the work takes place in the initBody method where we create a form and a hidden frame that will act as the target for the form. We also need to set the encoding, the method and the form action.
The corresponding FileUploadPage Presenter:
...
public class FileUploadPagePresenter extends
Presenter<FileUploadPagePresenter.MyView, FileUploadPagePresenter.MyProxy> implements
FileUploadUiHandlers {
// don't forget to update SerendipityGinjector & ClientModule
@ProxyStandard
@NameToken(NameTokens.fileUploadPage)
public interface MyProxy extends Proxy<FileUploadPagePresenter>, Place {
}
public interface MyView extends View, HasUiHandlers<FileUploadUiHandlers> {
}
@Inject
public FileUploadPagePresenter(EventBus eventBus, MyView view, MyProxy proxy,
PlaceManager placeManager) {
super(eventBus, view, proxy);
getView().setUiHandlers(this);
}
@Override
protected void onBind() {
super.onBind();
Log.debug("onBind()");
}
@Override
protected void revealInParent() {
RevealRootContentEvent.fire(this, this);
}
public static native void close() /*-{ $wnd.close(); }-*/;
@Override
public void onCancelButtonClicked() {
close();
}
@Override
public void onOkButtonClicked() {
close();
}
}
Now, lets take a look at the FileUpload Servlet:
...
public class FileUploadServlet extends HttpServlet {
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
this.process(request, response);
}
@Override
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
this.process(request, response);
}
private void process(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// check that we have a file upload request
if (ServletFileUpload.isMultipartContent(request)) {
processFiles(request, response);
}
}
private File tmpDir;
private static final String DESTINATION_DIR_PATH ="/files/upload";
private File destinationDir;
public void init(ServletConfig config) throws ServletException {
super.init(config);
DOMConfigurator.configure("log4j.xml");
Log.debug("FileUpload Servlet");
tmpDir = new File(((File) getServletContext().getAttribute("javax.servlet.context.tempdir")).toString());
if (! tmpDir.isDirectory()) {
throw new ServletException(tmpDir.toString() + " is not a directory");
}
String realPath = getServletContext().getRealPath(DESTINATION_DIR_PATH);
destinationDir = new File(realPath);
if (! destinationDir.isDirectory()) {
throw new ServletException(DESTINATION_DIR_PATH + " is not a directory");
}
}
private void processFiles(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// create a factory for disk-based file items
DiskFileItemFactory factory = new DiskFileItemFactory();
// set the size threshold, above which content will be stored on disk
factory.setSizeThreshold(1 * 1024 * 1024); // 1 MB
// set the temporary directory (this is where files that exceed the threshold will be stored)
factory.setRepository(tmpDir);
// create a new file upload handler
ServletFileUpload upload = new ServletFileUpload(factory);
try {
String recordType = "Account";
// parse the request
List items = upload.parseRequest(request);
// process the uploaded items
Iterator itr = items.iterator();
while (itr.hasNext()) {
FileItem item = (FileItem) itr.next();
if (item.isFormField()) {
if (item.getFieldName().equals("recordType")) {
recordType = item.getString();
}
} else {
// write the uploaded file to the application's file staging area
File file = new File(destinationDir, item.getName());
item.write(file);
// import the CSV file
importCsvFile(recordType, file.getPath());
}
}
}
catch (FileUploadException e) {
Log.error("Error encountered while parsing the request", e);
}
catch (Exception e) {
Log.error("Error encountered while uploading file", e);
}
}
After the file has been uploaded its is written to the application's file staging area and then it is processed by the importCsvFile method:
...
@SuppressWarnings("unchecked")
private void importCsvFile(String recordType, String filename) {
try {
// get the Class object for this recordType's DAO
Class dao = getEntityDao(recordType);
Object daoObject = createEntityDao(dao);
// get the Class object for this recordType
Class entity = getEntity(recordType);
Field[] fields = entity.getDeclaredFields();
// allow access to the Entities fields
AccessibleObject.setAccessible(fields, true);
// initialise the CSV Reader
CSVReader reader = new CSVReader(new FileReader(filename));
String [] nextLine;
// process the first line in the file
if ((nextLine = reader.readNext()) != null) {
// build the HashMap of Field names
getFieldNames(nextLine);
}
// process the file a line at a time
while ((nextLine = reader.readNext()) != null) {
persistEntity(daoObject, createEntity(entity, fields, nextLine));
}
}
catch (ClassNotFoundException e) {
Log.error("Error encountered while looking for class", e);
}
catch (FileNotFoundException e) {
Log.error("Error encountered while reading file", e);
}
catch (Exception e) {
Log.error("Error encountered while importing file", e);
}
}
private static final int DEFAULT_HASH_MAP_CAPACITY = 100;
private HashMap<String, Integer> fieldNames;
private void getFieldNames(String [] nextLine) {
fieldNames = new HashMap<String, Integer>(DEFAULT_HASH_MAP_CAPACITY);
for (int i = 0; i < nextLine.length; i++) {
// e.g. "accountName", 1
fieldNames.put(nextLine[i].trim(), new Integer(i));
}
}
@SuppressWarnings("unchecked")
private Object createEntity(Class entity, Field[] fields, String [] nextLine) {
Log.debug("createEntity()");
try {
Object object = entity.newInstance();
for (Field field : fields)
{
Class type = field.getType();
// ignore Static fields
if (Modifier.isStatic(field.getModifiers())) {
continue;
}
if (type.getSimpleName().equals("String")) {
Integer index = (Integer) fieldNames.get(field.getName());
if (index != null) {
field.set(object, nextLine[index].trim());
}
} else if (type.getSimpleName().equals("List")) {
List<Object> list = new ArrayList<Object>();
Field declaredField = object.getClass().getDeclaredField(field.getName());
Type genericType = declaredField.getGenericType();
if (genericType instanceof ParameterizedType) {
ParameterizedType pt = (ParameterizedType) genericType;
Type [] t = pt.getActualTypeArguments();
// e.g. "class au.com.uptick.serendipity.server.domain.Address"
String className = t[0].toString().substring(6);
Log.debug("className: " + className);
Class nestedEntity = Class.forName(className);
Field[] nestedFields = nestedEntity.getDeclaredFields();
AccessibleObject.setAccessible(nestedFields, true);
Object nestedObject = createNestedEntity(nestedEntity, nestedFields, nextLine);
if (nestedObject != null) {
list.add(nestedObject);
field.set(object, list);
}
}
}
}
return object;
}
catch (Exception e) {
Log.error("Error encountered while creating entity", e);
}
return null;
}
@SuppressWarnings("unchecked")
private Object createNestedEntity(Class entity, Field[] fields, String [] nextLine) {
Log.debug("createNestedEntity()");
try {
Object object = entity.newInstance();
for (Field field : fields)
{
Class type = field.getType();
// ignore Static fields
if (Modifier.isStatic(field.getModifiers())) {
continue;
}
if (type.getSimpleName().equals("String")) {
Integer index = (Integer) fieldNames.get(field.getName());
if (index != null) {
field.set(object, nextLine[index].trim());
}
}
}
return object;
}
catch (Exception e) {
Log.error("Error encountered while creating nested entity", e);
}
return null;
}
@SuppressWarnings("unchecked")
private Object createEntityDao(Class dao) {
try {
return dao.newInstance();
}
catch (Exception e) {
Log.error("Error encountered while creating entity dao", e);
}
return null;
}
@SuppressWarnings("unchecked")
private Long persistEntity(Object daoObject, Object entityObject) {
Class[] parameterTypes = new Class[] {Object.class};
try {
Method method = daoObject.getClass().getMethod("createObject", parameterTypes);
Object result = method.invoke(daoObject, new Object[] { entityObject });
return (Long) result;
}
catch (SecurityException e) {
Log.error("Security error encountered while persisting entity", e);
}
catch (NoSuchMethodException e) {
Log.error("Method not found while persisting entity", e);
}
catch (Exception e) {
Log.error("Error encountered while persisting entity", e);
}
return null;
}
@SuppressWarnings("unchecked")
private static Class getEntity(String name) throws ClassNotFoundException {
return Class.forName("au.com.uptick.serendipity.server.domain." + name);
}
@SuppressWarnings("unchecked")
private static Class getEntityDao(String name) throws ClassNotFoundException {
return Class.forName("au.com.uptick.serendipity.server.dao." + name + "Dao");
}
}
The importCsvFile method processes a CSV file a line at a time. The first line must contain the entities field names:
A sample CSV file in LibreOffice Calc:
5. Test the application in development mode
At this point, you should be able to compile Serendipity and launch it from within Eclipse.
The nested Accounts presenter and view (showing 5 accounts):
The nested Accounts presenter and view (after importing 5 new accounts):
Note: You can download the sample application from the CRMdipity (Se-r-en-dipity) project's download page.
What's Next
At this point, we've learnt how to add support for importing CSV files into Serendipity by taking advantage of Apache FileUpload and opencsv. Next we'll look at how to add login security to Serendipity.
- Tags: