In this post we are presenting a framework called automated sentiment analysis for appraisal data, which can assist people in making the appraisal review decision process more effective and efficient. This post is organized as follows:
- Section 1 gives background information about the appraisal data.
- Section 2 explains the motivation behind this work.
- Section 3 explains the automated sentimental analysis methodology.
- Section 4 gives a conclusion.
- Section 5 explains the applications of our work and this post will end with
- Section 6, an appendix of classifier performance.
In the U.S. real estate market, the single-family house is the most common type of residential property. For a lender to approve a loan, an appraisal has to be ordered on the property. Typically, the appraisal is done in the so-called “1004” form. In addition to collecting characteristic data about the property, appraisers also provide their subjective comments, i.e. their personal opinion. Below are some examples of appraisers’ comments. (These comments have been taken directly from the appraisal report).
- • C3;Kitchen-remodeled-one to five years ago, Bathrooms-remodeled-one to five years ago, THE CONDITION AND QUALITY OF CONSTRUCTION OF THE SUBJECT ARE AVERAGE AND CONSIDERED TO BE TYPICAL OF THE AREA. THERE WERE NO NEEDED REPAIRS NOTED.
• The subject property is a 3 Story attached three family dwelling with full basement. It is in overall average condition with sound brick exterior, dry foundation and average quality roof that had no reported leaks. The interior has standard kitchens and bathrooms with average quality cabinets and appliances, all in working order. No need for repairs as of the time of inspection.
• THE HOME APPEARS TO BE IN FAIR CONDITION BASED ON A DRIVE-BY APPRAISAL, REALTOR COMMENTS AND THE SUBJECT’S AGE. KNOWN DEFERRED MAINTENANCE INCLUDED SOME ROTTEN EXTERIOR SIDING ALONG BOTTOM EDGEG. HOME HAS A DETACHED BLOCK STORAGE BLDG THAT HAS ROOF LEAKS AND WAS GIVEN NO VALUE.
• C4;No updates in the prior 15 years;THE SUBJECT PROPERTY IS IN AVERAGE CONDITION. THE IMPROVEMENTS INCLUDING TILE FLOORING, CARPET, STANDARD KITCHEN APPLIANCES AND TILE COUNTERS IN ALL BATHROOMS ARE THE BASIC BUILDER UPGRADES. THERE IS PEELING EXTERIOR PAINT IN THE RONT LOWER AND UPPER LOORS.
The motivation behind this work is to design a system which can automatically classify the appraiser/user comments into one of the three categories, i.e., positive feedback, negative feedback and undetermined feedback and then assist people in making the appraisal review decision process more effective and efficient. The system will also help users find high-risk applications quickly.
We have designed our automated sentiment analysis system by utilizing the techniques offered by text mining and supervised learning. An outline of the system is shown in Figure 1.
There are three phases in the system. Below is a brief description of each phase.
Phase 1: Feature Extraction – Feature extraction is used to transform the raw textual data into a numeric set of features that can be fed to a classifier. These features set will extract the relevant information from the input data in order to perform the desired task. Figure 2 in the appendix section shows a snapshot of the transformed numeric data.
Phase 2: Feature Selection – Feature selection is used to select a subset of relevant features for use in model construction. In this phase we will find the top features from the features obtained in Phase 1. This has been done by using the ranker method and eliminates all the features/words whose frequency is below some threshold2. Figure 3 in the appendix section shows the snapshot of the trained data.
Phase 3: Classifier Training and Results – In previous phases we have preprocessed the raw data. The finished product, training data, will be used to train the classifier. We have experimented with different classifiers like Naïve Bayes, Random Forest, Neural Network, SVM, and SMO, but Naïve Bayes3 has outperformed all other classifiers and achieved the best classification accuracy. Therefore we have used a Naïve Bayes classifier for our automated system to classify the new instances into their respective category.
We have performed the experiments to evaluate the accuracy and efficiency of our automated sentiment system. The obtained results have been validated by using statistical measures such as 10-fold cross-validation4, TP-Rate and FP-Rate, etc. In this study, we have used an improved condition appraisal dataset, which consists of 491 instances. These instances have been labeled with the help of a reviewer. Below are the brief results of our experiments. We have used two versions of Naïve Bayes; one is Gaussian Naïve Bayes and other is Multinomial Naïve Bayes. Our automated system has been able to correctly classify 412 instances by using Gaussian Naïve Bayes and 407 samples using Multinomial Naïve Bayes. Detailed results are presented in Section 6.1.
The manual appraisal review decision process takes up a significant amount of time and can be cumbersome, which can lead to poor efficiency and diminished performance. In order to improve this manual system, we have designed an automated sentiment system, which is effective and efficient and can overcome the existing limitations. The applications of our system are discussed in Section 5.
Once the classifier is developed like Naïve Bayes in Section 3, it is embedded in the regular program and can be used for different applications. Below are some of the applications.
5.1 Live Determination
The first application is live determination to determine the category of each appraisal comment. We designed a user interactive program which is a two-step process. The first step is an input step to enter appraisal comments. In the second step our sentiment automated system will break the input comments into different sentences, determine the sentiment polarity and predict the category for each comment. For example:
- (User Input): The kitchen appears to have been updated within the last 3-8 years. All windows except for those found at the family room addition at the rear have been replaced with thermal pane windows that appear fairly new. The subject appeared to be in average condition from the exterior with respect to other properties in the subject development and subject market area. The backyard condition and interior condition obviously couldn\’t be determined during the “drive-by,”. The subject’s exterior wood trim is in need of scraping/painting where peeling or chipping has occured. The estimated cost to cure is approximately $750.00 or less. This does not have a negative impact on the subjects marketability nor the Appraisers opinion of value for the subject property. Kitchen feature wood cabinets, laminate countertops and vinyl flooring,updated bathroom feature ceramic tile flooring and ceramic tile shower enclosure.
Our automated sentiment system will give output in the following way. (Red represents Negative feedback, Green represents Positive feedback and Blue represents Undetermined feedback.) The system will assign a category to each type of feedback with their probability values and will also tell the percentage of their presence in other categories in terms of probability values. Below is the system output of the above user input.
5.2 Batch Mode
The second application is batch mode. Consider this hypothetical question: Are the appraisals from two areas having the same sentiment distribution? In other words, would it be possible for one area’s appraisals to have more positive (or negative) comments than the other one? A batch mode process can facilitate this analysis. In the below example, we took a pool of appraisals consisting of 4,000 paragraphs. Our system broke this down into 15,732 sentences and assigned all of them their sentiment polarity, which consists of 2,343 sentences in the Positive category; 2,060 sentences in the Negative category and 11,329 sentences in the Undetermined category.
In this section we have discussed the results in a detailed manner and have presented the figures used in our methodology.
We discussed the brief results in Section 3 (Methodology). Below is their detailed explanation. In this study, we used an improved condition appraisal dataset consisting of 491 instances. It has 94 instances in the Positive category; 107 instances in the Negative category and 290 instances in the Undetermined category. These instances have been labeled with the help of a reviewer.
6.1.1 Gaussian Naïve Bayes Classifier
When our system used the Gaussian Naïve Bayes classifier, we able to achieve 83.91% overall accuracy where 412 instances were correctly classified and 79 instances were wrongly classified to other categories. The Positive category had 94 instances out of which 71 were correctly classified and 23 were wrongly classified to the Undetermined category. The Negative category had 107 instances out of which 67 were correctly classified, 2 were wrongly classified to the Positive category and 38 were wrongly classified to the Undetermined category. The Undetermined category had 290 instances, out of which 274 were correctly classified and 8 each were wrongly classified to both the Positive and Negative categories. These results are shown in the table below.
6.1.2 Multinomial Naïve Bayes Classifier
When our system used the Multinomial Naïve Bayes classifier, we were able to achieve 82.89% accuracy overall, where 407 instances were correctly classified and 84 instances were wrongly classified to other categories. The Positive category had 94 instances, out of which 74 were correctly classified and 20 were wrongly classified to the Undetermined category. The Negative category had 107 instances, out of which 68 were correctly classified, 3 were wrongly classified to the Positive category and 36 were wrongly classified to the Undetermined category. The Undetermined category had 290 instances, out of which 265 were correctly classified, 7 were wrongly classified to the Negative category and 18 were wrongly classified to the Positive category. These results are shown in the table below.
Figure 2 is the snapshot of the transformed numeric data that we generated in Phase 1 using feature extraction method and Figure 3 is the snapshot of the training data that we generated in Phase 2 using feature selection. This training data is fed to Naïve Bayes to train it.
4 http://en.wikipedia.org/wiki/Cross-validation_ (statistics)
6.4 Suggestions and Comments
Through data mining techniques, we were able to make an automated system for appraisal data that is efficient and effective and can overcome all the existing limitations of a manual process.
In closing, thanks for spending your valuable time reading this post. Please do send your suggestions and comments about our work.